Evidence-based policy making practice
based on the international evaluation
of funding and management of NSFC
Zheng Yonghe
Deputy Director General, Bureau of Science Policy, NSFC
Kunming, May 23, 2014
Outline
 NSFC’s funding policy development: Dimension discussion
 The basis of policy making: Evidence-based evaluation
 Essential work of evaluation: Evidence collection
 Case study: International evaluation of funding and
management of NSFC and the policy making
 NSFC’s funding policy development: Dimension discussion
Positioning
• Strategic positioning in NIS of China
• Promote basic research development
• Fostering talent
• International cooperation
How it works
• Modality of funding management
• Funding instruments
• Peer review system
Performance
•
•
•
•
Contribution to science development
Contribution to talent training
Contribution to socioeconomic development
Impact of NSFC’s funding
Principals: Problem-driven; reform-driven
 The basis of policy making: Evidence-based evaluation
The national performance evaluation system has not established in China,
currently. NSFC’s international evaluation on funding and
management performance provides a typical success case for
comprehensive evaluation in China.
1.
Evaluation as an research-based approach plays an important role in
policy making.
2.
Evidence-based evaluation insure the legality and credibility of
policy in policy argument process.
3.
Question-oriented evaluation and independent third-party evaluation
are more feasible and convincing.
 The basis of policy making: Evidence-based evaluation
Suggestion
Suggestion
Directly goes to
policy making
After further
research, goes to
policy making
Short-term
Medium-term
Long-term
The pathways and period of suggestion into policy
 Essential work of evaluation: Evidence collection
Evidence-based
Conclusion of
the Evaluation
• Suggestions based on
evidence
Evidence-based
Implementation of
the Evaluation
• Collecting evidences
from data, information
& activities
Evidence-based
Design of
the Evaluation
• Designing evaluation
plan based on evidence
 Case Study: International evaluation of funding and
management of NSFC and the policy making
1. Brief overview of international evaluation of funding
and management of NSFC
 The evaluation is commissioned by MOF and NSFC
together. The State Council attached importance to this
evaluation.
 Dual Objectives of the Evaluation:
– Accountability: Provide an independent assessment of the
overall performance of NSFC’s funding and management during
the past 25 years.
– Lesson learning: Improve the NSFC’s funding and management
performance; redefine NSFC’s strategic role within the NIS of
China
Brief overview of international evaluation
of funding and management of NSFC
 Evaluation Approach and Principals
– Modality: Domestic preparation followed by international evaluation
– Approach: Evidence-based
– Principals: Independent, Objective
 Evaluation Framework
Based on the dimension discussion
Dimensions
NSFC’s strategic
positioning
Funding
performance
Management
performance
Impact on China’s
S&T system
Issues
Questions
1 NSFC’s strategic positioning in the national
innovation system of China
Q1-Q3
2 Funding strategy
Q4-Q6
3 Contributions to original innovation
Q7-Q8
4 Promoting the balanced development of scientific
disciplines
Q9-Q11
5 Fostering innovative talents
Q12-Q14
6 Supporting the national demands and challenges
Q15-Q17
7 Modality of funding management
Q18-Q20
8 Funding instruments
Q21-Q23
9 Peer Review System
Q24-Q26
10 Impact of NSFC’s funding
Q27-Q30
Brief overview of international evaluation
of funding and management of NSFC
 Functions of Actors in the evaluation
Synthesis evidence report:
Provided by independent
evaluation organization
NCSTE
IEC
Typical cases:
Provided by NSFC
NSFC
International evaluation report
Brief overview of international evaluation
of funding and management of NSFC
 Implementation of the Evaluation
Phase
Major Evaluation Activities
Phase1: 2010. 2－2010. 6
Inception and evaluation design
Phase2: 2010. 7－2010.12
Prepare evaluation evidence report
The IEC first meeting in Beijing
Phase3: 2011. 1－2011. 6
Evaluation activity of IEC
Complete the Evaluation Report
2. Evidences and the evaluation procedure
scheme design
Designing evaluation
Typical cases
Cases
report
Thematic research
Literature
metrology
report
Thematic
report
Collecting evidences
Survey
report
Interview
report
Synthesis evidence report
Domestic preparation
Individual Evaluating
by expert
Focused evaluating
by Expert Panel
International evaluation
第12页（共25页）
International
evaluation report
Writing report by
Chair & rapporteur
Special
report
A. Designing evaluation:
Factor analysis in designing framework based on evidence
factor: evidence
DFG
RCN
JSPS
FWF
SGCRP
NSF
FP6
Backgroud
■
Objectives
Framework
■
■
■
■
Information source
■
■
■
■
■
Subject of assessment
■
■
■
■
■
Procedure & Time
■
■
Performance report
Application of results
Budget
■
■
■
Director
of NSFC
■
■
■
Commissioning entity
& Organizers
■
Interview
■
■
■
Literature of
evaluation
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
B. Domestic preparation: collecting evidences from
questionnaire surveys, interviews and other activities
Six sets of questionnaire surveys: delivered 77799, returned 20221
Delivered
Returned
Respond
Rate
Institutes and Universities
2258
841
37.25%
PIs of the General Program projects
43412
10228
23.56%
Grantees of Young Scientist Fund projects
16092
6104
37.93%
Grantees of the National Science fund for Distinguished
Young Scholar
2164
445
20.56%
Reviewers of proposals
6036
1232
20.41%
Rejected applicants
7838
1371
17.49%
Total
77799
20221
25.99%
Questionnaire survey
B. Domestic preparation: collecting evidences from
questionnaire survey, interview and other activities
Interviews: Focus group meeting & face-to-face interview 53times, involving 294 persons
Types of Interviews
NO.
NO. of participants
Research managers from institutes and universities
7
73
Researchers from institutes and universities
11
94
PIs of NSFC projects
3
24
Academic leader of related discipline
1
4
Rejected applicants
6
42
Reviewers of proposals
1
11
Staff of NSFC
13
33
S&T management officials at central government level
1
3
Academicians of CAS and CAE
6
6
Former and current presidents and vice-presidents of NSFC
4
4
53
294
Total
C. International evaluation: From ‘Synthesis evidence
report’ to ‘International evaluation report’
IEC presents the evaluation results in ‘International evaluation report’ based on
‘Synthesis evidence report by NCSTE individually.
NCSTE
IEC
 Case Study: International evaluation of funding and
management of NSFC and the policy making
3. Evidence-based policy making: After Evaluation
 Case 1: Raising ‘Excellent Young Scientists Fund’
 Case 2: One year ‘Time out’ for applicants who have
been unsuccessful in two consecutive years.
Case 1: Raising “Excellent Young Scientists Fund”
based on age distribution of DYS fund
 Evidence
Naturally, having fostered young scientists in its talent programs
NSFC is now seeing them age. The effect is especially marked in the
DYS found, where increasingly those just below the age limit win the
grants.
Case 1: Raising “Excellent Young Scientists Fund”
based on age distribution of DYS fund
 Suggestion from ‘International evaluation report’:
We recommend that a new program should be established
for younger researchers, no longer based on biological age
but rather on ‘scientific age’……
and, recommend that a substantial ‘China Chair’ program
(in the style of the Canada Research Chairs) be established
by NSFC for junior investigators with sustained support
over years……
Case 1: Raising “Excellent Young Scientists Fund”
based on age distribution of DYS fund
Type: Directly into policy
Evidences source: Data statistics, Evaluation report
 New policy making of NSFC
Raising “Excellent Young Scientists Fund” in 2012
The ‘Excellent Young Scientists Fund’ plans to support with funding of 1
million RMB per project.
Case 2: One year ‘Time out’ for applicant who has been
unsuccessful in two consecutive years.
 The challenge of NSFC in 2009
More management pressure on NSFC staff: The growth in applications
has been much faster than the growth in staff in recent years. The annual
number of applications per permanent member of staff has risen from 142
in 2001 to 505 in 2009.
More pressure on Reviewers comparing with NSF or DFG.
More pressure on Reviewers
Proposal Staff
NSF
Reviewer
45000
1400
42000
NSFC
119000
200
40000
DFG
35000
750
9000
Case 2: One year ‘Time out’ for applicant who has been
unsuccessful in two consecutive years.
 The challenge of NSFC in 2009
Rapid increasing proposals of GP: As the main instrument of NSFC,
the General Program application has been increasing rapidly since 2002.
？
the future?
Case 2: One year ‘Time out’ for applicant who has been
unsuccessful in two consecutive years.
 Alternatives of Policy
1
Limiting the proposals from institutes and universities
2
3
Enforcing a ‘time out’ for applicants who have been
unsuccessful in consecutive years
Adjusting the grant size and duration of General Program
How to make the decision?
Evidence-based evaluation
Policy making
Case 2: One year ‘Time out’ for applicant who has been
unsuccessful in two consecutive years.
 Step 1: Analysis the reason of rapid increasing
 NSFC grants are increasingly attractive to actual and potential beneficiaries.
 Universities and other organizations are using receipt of NSFC funding as an esteem
indicator for promotions and in the assessment of large projects, further increasing
application pressure.
Evidence: Questionnaire survey of supporting institutions
• 75% of respondents use NSFC grants as an important basis for various merit-based
assessments of research staff.
• 58% of respondents regard NSFC grants as a condition for job promotion or assignment.
• 56% of respondents allocate additional research funds to NSFC projects.
• 51% of respondents pay cash bonuses or other benefits to NSFC grantees.
• Only 8% of respondents have no special policies for NSFC awards or awardees.
Case 2: One year ‘Time out’ for applicant who has been
unsuccessful in two consecutive years.
 Step 2: Evaluation based on the evidence of stakeholder
surveys and data analysis
 Stakeholder 1: Staff of NSFC
a) When interviewed, the staff from the Science departments complain
that “The current workload is too heavy, and the only time we can
have for ourselves is the time for sleep.”
b) Most of the staff who attended the group meeting also complained
about the overloading. One of the participants pointed out that a
program director had only eight minutes to review a proposal in the
given timeframe.
Case 2: One year ‘Time out’ for applicant who has been
unsuccessful in two consecutive years.
 Stakeholder 2: Institutes and universities
a) Most institutes and universities oppose to limit the number of
applications, because more conflicts will emerge between applicants
and institutions, and more and more applicants will get the
opportunity by “old-friend network”.
b) 43.5% of 841 institution respondents considered the current success
rate of application for the GP was appropriate, while 46.0%
considered it moderately low or low.
Case 2: One year ‘Time out’ for applicant who has been
unsuccessful in two consecutive years.
 Stakeholder 3: PIs of General Program
The results from the questionnaire survey of PIs of the GP show that,
53.8% of 10288 respondents considered the grant size of GP was
appropriate, while 43.5% considered it was less than the actual research
costs. Only 0.6% of the respondents thought it was more than actual
research costs. The rest 2.1% of respondents’ answers to the question is
“I’m not sure”.
Case 2: One year ‘Time out’ for applicant who has been
unsuccessful in two consecutive years.
 Stakeholder 4: Reviewers
45.5% of reviewer respondents thought the current success rate of the GP
was appropriate, while 49.8% thought that it was moderately low or low.
Case 2: One year ‘Time out’ for applicant who has been
unsuccessful in two consecutive years.
 Stakeholder 5: Rejected Applicants
44.1% of rejected applicant respondents agree that applicants who were
consecutively denied for two years are not permitted to submit proposal
in the third year, but can submit proposal in the fourth year.
Case 2: One year ‘Time out’ for applicant who has been
unsuccessful in two consecutive years.
• Evaluation suggestion:
At least one year time our for applicant who
has been unsuccessful in two consecutive
years.
Case 2: One year ‘Time out’ for applicant who has been
unsuccessful in two consecutive years.
 Step 3: Further research for policy making after evaluation
General Program(GP)
22%
20%
资助率：19.24%
2.85% 18%
16.39%
15.43%
16%
14.82%
14%
12%
N=2
N=3
N=4
Young Scientists fund(YSF)
30%
28%
-3.35%
27.92%
 The success rates of GP after 2-4
consecutive rejected years are lower
than the average rate.
 But the success rates of YSF and LDR
after 2-4 rejected consecutive years are
higher than the average rate.
 The trend will be more obvious
following the increasing of consecutive
rejected years.
Fund for Less Developed Region(LDR)
28.88%
30%
28%
26.80%
26%
26%
24%
24%
23.88%
23.38%
24.17%
资助率：23.45%
22%
22%
20%
20%
N=2
N=3
N=4
-1.92%
资助率：21.96%
N=2
N=3
N=4
Case 2: One year ‘Time out’ for applicant who has been
unsuccessful in two consecutive years.
 Step 3: Further research for policy making after evaluation
100%
50000
 The proportion of applicants
who have been unsuccessful in
two or above consecutive years
maintains 70%- 80% in recent
years.
 The prediction said that it would
decrease about 30000 proposals if
NSFC applies a ‘time-out’ policy
to forbid the proposals of
applicants who were rejected in
both 2011 and 2012. (The
potential applicants proportion
was proposed 80%)
40000
70.22%
72.96%
75.52%
77.92%
79.83%
37253
80%
68.12%
31369
61.46%
30000
60%
53.04%
24660
22076
18036
20000
15108
40%
14380
12144
10000
25042
8646
13159
6071
8272
8014
8838
2006
2007
2008
16672
20%
19214
0%
0
2005
申请减少量
2009
未申请部分
2010
2011
2012
N次失败后申请率
2013
Case 2: One year ‘Time out’ for applicant who has been
unsuccessful in two consecutive years.
 Step 4: Policy making & performance of the new policy
 Policy making:
a) Enforcing one year ‘time out’ for applications who have been
unsuccessful in two consecutive years since 2014. NSFC had
informed the message to all potential applicants in 2013.
b) Adjusting the grant size, and the duration of General Program from 3
years to 4 years.
Case 2: One year ‘Time out’ for applicant who has been
unsuccessful in two consecutive years.
 result of the new policy:
The number of the whole proposals is obviously decreasing in 2013
and 2014 because of the ‘time-out’ policy.
Case 2: One year ‘Time out’ for applicant who has been
unsuccessful in two consecutive years.
 result of the new policy:
In contrast to the growth trend, the number of proposal for the GP
began decreasing dramatically in 2013.
Case 2: One year ‘Time out’ for applicant who has been
unsuccessful in two consecutive years.
 result of the new policy:
The grant size and success rate of GP began increasing obviously since
the implementing of the new policy.
The grant size and success rate of General Program in 2009-2013
Thanks！