Division of Molecular and Cellular Biosciences
Directorate for Biological Sciences
National Science Foundation
Response to the report by
Committee of Visitors (2011-13)
The Directorate for the Biological Sciences (BIO) and the Division of Molecular and Cellular
Biosciences (MCB) appreciate the hard work, insightful comments, and constructive
recommendations of the Committee of Visitors (COV) while assessing the processes and portfolio
management in MCB during FY2011-13. After receiving the COV report, the Division held a series of
discussions to develop an implementation strategy and found the recommendations made by the
COV to be thoughtful. This document contains the BIO response to specific recommendations1
made by the committee in its report.
General recommendations:
1. MCB must continue to provide an intellectual hub for key foundational research centered at
quantitative, predictive, theory-driven biology at the molecular and cellular level.
This approach is vital to the advancement of the entire field of Biology. The proactive outreach and
connections to other fields spawns new emerging areas of research. The co-funded interdisciplinary
projects are an excellent complement to the important single-investigator research grants. This is a
funding model of how NSF should work.
Response:
The Division agrees with the recommendation and will continue to give high funding priority to the
quantitative, predictive and theory-driven research in molecular and cellular biosciences. This will
be achieved through budgetary considerations and catalytic investments in the emerging areas at
the intersection of biology and other disciplines.
Sponsoring workshops in leading edge topics is one mechanism to ensure that MCB continues to be
an intellectual hub for key foundational research centered on quantitative, predictive, theory-driven
biology at the molecular and cellular levels. The Division believes that a clear plan to achieve
outcomes such as well publicized reports and publications is essential for a successful workshop. A
small list of workshop ideas will be developed based on the COV’s suggestions and internal budget
development discussions. These will be sponsored to obtain community input into budgetary
priorities and the Division’s portfolio development strategies.
2. Seeking transparency for Division and Directorate vision, process and policy.
1
The COV report summarized the major recommendations in the executive summary as well as additional
recommendations in the body of the report. The two sets of recommendations are largely, but not completely,
overlapping. In this document, the Division has first responded to the recommendations in the executive summary and
has added a section ‘other recommendations’ to discuss the additional recommendations that were made throughout
the body of the report.
1
The COV became aware of concerns about morale within the Division. The COV has confidence in
the management structure of MCB, and values the fundamental and multi-disciplinary program that
MCB has developed via the portfolio of funded projects. The COV perceives a threat to this valuable
program if morale in the Division becomes a distraction. There appears to be many recent and
imminent changes in the organization; the COV is concerned about how these changes will impact
the scientific community. The COV recommends greater transparency in the mechanisms used to
recruit and fill key management positions in BIO (e.g., Division Director, Deputy AD, AD positions).
Morale and HR inconsistencies were noted by the administrative staff. Staff were proud of their
performance and of the Division and its management; however, they feel as though their efforts go
unrecognized by the Directorate as a whole and would benefit from more transparency in policies
and hiring practices. Many of the staff do not have a clear sense of their career development
opportunities or potential for upward mobility.
Response:
The Division agrees that transparency in the Division and Directorate vision, processes, and policies
is critical in efficient and effective functioning of a Division towards accomplishing NSF’s mission of
advancing the progress of science. The Division values transparency, as it pertains to maintaining a
working environment grounded in openness, communication, and accountability. The Division
understands that transparency can be undermined in times of change, especially if staff perceive
there is not enough sharing of information about the outcomes—good and bad—of the change.
Therefore, the MCB leadership will increase its efforts to communicate information in the Division.
Currently MCB uses several methods to implement transparency in the Division, including: monthly
staff meetings, the ‘FYI MCB’ emails to the staff when there are some important developments, and
staff input in hiring process. In the recent Division retreat, we started implementing the COV
recommendation by including open discussions on the organization of administrative staff, and to
enhance transparency further, the Division will undertake the following actions:
1. Implementing additional regular communication: Currently the Division has a monthly staff
meeting for providing updates to and receiving input from the staff. It will experiment with
increasing the frequency of these meetings. All communications and standard operating
procedures will be archived on a SharePoint site to make them easily accessible to all staff.
2. Streamlining communication responsibilities: Currently many persons are involved in
internal and external communication (AAAS Fellow, Division Director, Deputy Division
Director, Program Support Manager, etc.). The Division will streamline both external and
internal communication by making it the primary responsibility of one program
specialist/biologist or program analyst/biologist in the Division. This individual will report to
the Deputy Division Director who will oversee the development and implementation of
communication strategies for MCB.
3. Changes in personnel practices: The Division strives to be open and fair in its recruiting and
hiring practices. MCB will increase transparency in the personnel processes through the
following steps.
2
a. When making offers to prospective candidates, the Division will clearly communicate
the opportunities that exist for career advancement, thus providing realistic
expectations.
b. Regular updates on recruitment efforts will be provided to the Division even if there
has been no progress.
At the Directorate level, transparency will be further enhanced in future hires and appointment of
Deputy Division Directors, Division Director, Deputy AD, and AD positions through the development
and communication of standard operating procedures by the permanent BIO Senior Executives.
3. Generate mechanisms to foster interactions among the divisions of the BIO Directorate. The
entire field covered by the BIO directorate is moving towards more quantitative and predictive
science. More coordination and interaction within MCB would help all the divisions better realize
their full potential. Mechanisms should be developed to provide incentives for collaborative funding
among Divisions of the BIO Directorate.
The COV observed a general lack of cohesion and integration between Divisions of the BIO
Directorate, and also among some clusters within MCB. In particular, Genetic Mechanisms Division
seemed disconnected and not coordinated with the rest of the MCB team.
The COV recommends a regular strategic planning meeting involving PDs, DDs and ADs across the
Bio Directorate in order to develop better cohesion and coordination of Bio funding activities. This
will provide the opportunity to better integrate the Bio Directorate team.
Response:
The Division agrees that the interactions among BIO divisions are critical for investing in the
transformation of biology from a descriptive science to a predictive science. The COV questions in
the standard report template did not refer to the interactions among the divisions within a
discipline. Therefore, data were not provided to the COV regarding the extensive interactions
between MCB and other divisions in BIO. During the period under review, MCB has collaborated
with each of the three other divisions for special activities as well as for supporting regular research
grants. Here are the examples of MCB’s collaborations with other BIO divisions.
IOS:
•
•
•
DEB:
•
DBI:
•
NSF-BBSRC Ideas Lab on Enhancing Efficiency of Photosynthesis and continuation of the
Ideas Lab projects
NSF-BBSRC Ideas Lab on Improving Nitrogen fixation and utilization
NSF-JST program on metabolomics and continuation of the funded projects
Workshops to develop a research agenda on the ecological impacts and evolutionary
implications of synthetic biology
NSF-BBSRC Lead Agency agreement for evaluation of US-UK collaborative projects in
computational biology, system biology, and bioinformatics
3
The newly developed portfolio management process in the Directorate contains opportunities for
the Program Directors to plan strategic collaborations for managing the exciting science frontiers in
their portfolios. These increased opportunities for interactions and for sharing information will help
to further increase the inter-divisional interactions in the Directorate.
MCB has provided budgetary incentives for aligning the portfolios of all clusters with the Division’s
consensus vision, while providing enough flexibility to maintain diversity of topics and systems. It
will continue to provide budgetary incentives for funding research projects that align with the NSF,
Directorate, and Division vision and priorities.
4. Ease Domestic and Foreign Travel Restrictions.
The current restriction on travel budget is a detriment to the overall well-being of the MCB program.
The PDs need to continue to have opportunities to travel to national and international meetings, and
for other outreach activities as well as to various institutions to interact with funded PIs and with
potential future PIs.
The ability of MCB program leaders to remain active participants in the greater scientific community
is crucial to their effectiveness in managing the competitive review process, but more so in their
ability to make quality judgments on proposals. Undue travel restrictions on MCB program leaders
will jeopardize portfolio balance.
Response:
The Division agrees that restrictions on Program Director travel (imposed by OMB guidelines) have
the potential to undermine the Division’s ability to meet its scientific goals and to serve its PI
communities. To implement the COV recommendation, prioritization of domestic and foreign travel
will continue to be determined by the Division, because the Division is in the best position to
understand the potential outcomes of the travel.
5. Generating a plan for broadened career development for MCB trainees.
The COV is concerned about the general challenge presented by the limited opportunities for Ph.D.
and postdoctoral scientists who are being trained with NSF support, and the evolution of the
scientific workforce and its diversity in the United States. How are MCB and NSF responding to
the Alberts et al. PNAS paper noting that only 8% of Ph.D. trained-scientists go to academic
positions?
Response:
The Division believes that the issues in the Alberts et al. paper should be addressed where
appropriate. The focus of this analysis was on biomedical research and training, and as these areas
are not supported by MCB, the issues may not be as pressing for the Division’s stakeholder
community. However, in the next version of the core program solicitation, the Division will
encourage PIs to consider the findings of this paper in the mentoring of students and post-docs in
individual research projects, recognizing that many might choose careers beyond academia. The
Division will also consider presenting information about alternative science careers in the
Foundation during outreach to the community.
6. Evaluation, documentation and follow up on Broader Impacts
4
We recommend that applicants and reviewers be instructed to address both aspects of Broader
Impacts:
a) What are the broader scientific and societal impacts of the proposed research, and
b) What are the broader impacts associated with the supported personnel, such as education and
outreach activities?
This will encourage better uniformity in the review process and level the playing field for applicants.
The Broader Impacts of the project to society should always be explicitly addressed and will aid in
justification of awards to the general public.
Finally, annual reports and the Results of Prior Support sections of proposals should clearly outline
progress towards the stated Broader Impact goals to ensure follow up.
Response:
MCB agrees that the broader impacts criterion has to be explained better to the reviewers as well as
to the PIs for a consistent evaluation of the criterion. MCB will continue to provide guidance to the
PIs and reviewers about the broader impacts definition as stated in the Grant Proposal Guide. The
Division has started to consider a number of actions for improving review and evaluation of broader
impacts. Based on the previous COV’s recommendation, MCB has instituted a reporting
requirement for educational and training broader impacts. Now that the annual reports for the
projects that were funded since this requirement was instituted are available, the Division will
assess the effectiveness of this requirement in evaluating the outcomes of the broader impacts
activities of MCB-funded projects.
7. Develop and Improve methods for communicating scientific achievements and the value of
fundamental science.
Without basic science there will be no future translational research. This is a particular, but
important, challenge for foundational areas such as those funded by MCB. An avenue should be
identified to better publicize the highest quality scientific and educational highlights of funded
projects.
Response:
MCB agrees that communicating the exciting outcomes of MCB funded research is an important
responsibility of the Division. In response to this recommendation, MCB will dedicate one FTE as a
Biologist/Program Specialist for internal and external communication. MCB has already started
discussions about a communication strategy for the Division, which will be finalized in FY15.
8. Evaluation of the use and review of EAGER awards. The COV has concerns about the mechanism
and use of EAGER grants. Often they are not used for their intended purpose. Evaluations and
funding decisions appear to be made primarily by a single PD. The COV suggests that a more
inclusive and transparent mechanism be developed, with evaluation by at least 2 PDs documented in
the Review Analysis. The basis/rationale for using the EAGER mechanism should be made clear in the
documented reviews. Other means for bridging funds on closed grants need to be developed.
Response:
5
MCB thanks the COV for its recommendation about the EAGER awards. The Division agrees that
EAGERs should be used for their designated purpose - to support the early stages of potentially
transformative (high risk-high payoff) ideas. To address the concerns of the COV, provide
transparency and accountability, and ensure a level playing field for all PIs, MCB will implement the
following actions:
•
At least two Program Directors in addition to the cognizant Program Director will be involved
in the review of EAGERs and the comments from them will be recorded appropriately.
•
The Division Director will ensure that the cognizant Program Director has adequately
addressed the criteria for EAGERs.
•
The high-risk nature of EAGERs means that some failure should be expected. In such cases,
the cognizant Program Director should ensure that the Final Project Report includes an
explanation about why the project was not successful in accomplishing its goals.
The Division would also like to share the results from a just-completed study by a summer intern
who examined 49 EAGERs funded by MCB since 2009. The intern found that EAGERs had equal or
better outcomes compared with peer-reviewed awards, as judged by the numbers of patents,
publications, and citations. After the expiration of EAGER awards, some PIs continued their projects
while others did not. MCB considers these to be signs of a relatively healthy approach to funding
high risk – high reward ideas, and believes the above changes in process, implemented in response
to the COV's report, will strengthen it.
9. Avoiding unnecessary conflict of interest (COI) restrictions:
We recommend that the terms, restrictions and definition of COI be re-evaluated and updated. The
COV is concerned that with the increased emphasis and funding for large collaborations and
interdisciplinary projects, the 48-month restriction of coauthoring a paper will make it difficult to
find qualified reviewers who do not have COI. A careful analysis of multi-PI publications might help
to distinguish between close collaborations and one-time community projects.
Response:
MCB will communicate this recommendation to the Office of General Counsel, which is responsible
for issuing COI rulings. It should be noted that when a paper contains dozens of authors (e.g.
genome sequencing papers), the Office of General Counsel has routinely waived COI for most of the
authors.
10. Continue to implement the once per year submission cycle for proposals.
The COV appreciates the rationale for the recent implementation of a single annual submission date.
The COV strongly believes that the pre-proposal mechanism should not be implemented by MCB.
Response:
As recommended by the COV, MCB intends to continue to accept full proposals annually and does
not plan to implement the pre-proposal mechanism at this time.
6
Other recommendations
In the COV report, there were additional suggestions in the sections in the COV report template
(listed below). We will address these suggestions these suggestions or communicate them to
appropriate offices at NSF (e.g. Division of Information Systems).
•
We further encourage the Division to continue to seek solutions to the problem of
promoting substantive reviews without prescribing a "checklist".
Response:
The Division agrees with this recommendation; MCB will emphasize that program directors
should provide panelists with guidance on writing reviews that discourages the use of checklists.
MCB will investigate ways (e.g. through modifying the invitation letter) to encourage ad hoc
reviewers to write substantive reviews.
•
.
We urge that, when Program Directors ask potential reviewers whether they are willing to
review a particular proposal, at that stage only the Project Summary should be provided.
Access to the entire proposal should only be given to reviewers who actually agree to review
the proposal.
Response:
The Division agrees that a system in which a reviewer was given access to the full proposal only
after agreeing to review would be desirable. However, formal requests to review proposals are
sent to potential reviewers using the NSF electronic correspondence system, because this
system requires potential reviewers to acknowledge and accept the conflict of interest rules
before they access the proposal. Unfortunately this system for assigning and generating review
letters cannot generate review requests that give access only to the abstract. The suggested
change could not be implemented manually within the current system, therefore, this
recommendation will be forwarded to the Division of Information Systems.
•
We have a technical suggestion that will help future COVs: In the spread sheet with the data
on the proposals, the Average Merit score is calculated excluding the reviews that give
multiple ratings. We wondered why proposals with average scores of 5.0 were declined
while proposals with an average score of 3.0 were funded. Upon examination, the former
situation arises when a proposal is rated E by one reviewer and G/F by two other reviewers,
for instance. We feel that the dual scores should be counted as half-integer scores; thus in
this example the average score should be (5 + 2.5 + 2.5)/3 = 3.3, not the 5.0 that is currently
reported. The latter situation is illustrated by another example, wherein a proposal that is
scored as G by one reviewer and E/V by three reviewers should be averaged as: [3.0 + (3 x
4.5)]/4 = 4.1 and not 3.0 as reported.
Response:
MCB agrees with this recommendation. This shortcoming will be reported to the Division of
Information Services.
7
•
We think in-presence reviewing is a good practice that should be continued to ensure an
open discussion and consensus decisions.
Response:
MCB agrees with recommendation and will use in-person panels as much as possible.
•
The COV however, is concerned that the PIs of declined CAREER proposals may not have
sufficient time (6 weeks) to resubmit to the regular mid-November deadline, since the way a
CAREER proposal is put together is very different from a regular proposal.
Response:
MCB agrees with COV’s concern and will consider it in the next revision of the core program
solicitation.
•
It would have been helpful to provide data segregated by award type to evaluate the size
and duration of projects, especially the conference awards, as these tend to be small in size
and duration, skewing the aggregate data.
Response:
MCB agrees with recommendation and provide data segregated by award type to the next COV.
8