Knowledgeable Bankers?
The Demand for Research in World Bank
Operations
Martin Ravallion
Knowledge matters to the quality of
development aid
• Stock of Bank analytic work on a recipient country is a strong
predictor of the quality of its lending operations (Deininger,
Squire and Basu, 1998; Wane, 2004).
• And the quality of prior analytic work matters to the quality of
the Bank’s projects (Fardoust and Flanagan, 2011).
• The generation of knowledge and its diffusion within donor
agencies is a poorly understood factor in development
effectiveness.
2
The market for knowledge:
Both supply and demand matter!
• We often focus on the supply side, which is clearly important.
• We try to assure that:
– knowledge products are both rigorous and relevant.
– we have the right skill mix for knowledge generation.
– we are in the right partnerships.
• But this will all come to nothing if the incentives to learn—the
demand side—are weak.
• The incentives for learning within aid organizations have been
identified as one factor in the quality of aid (Wane, 2004;
Ravallion, 2011).
3
Yet we know very little about the demand for
the Bank’s research among operational staff
First set of questions addressed by this paper:
• How familiar are the Bank’s practitioners with the Bank’s
research?
• How do the answers to these questions vary across units
and sectors of the Bank?
4
We know even less about the incentives for
learning among the Bank’s operational staff
• Those incentives depend crucially on the value attached to
WB research by staff for their work.
Second set of questions addressed by this paper:
• Do the Bank’s practitioners value research for their work?
• And (if so) does this incentive to learn translate into greater
familiarity and use of the Bank’s research?
5
A typology of staff by knowledge and
incentives to learn
Incentive to learn:
Perceived value of Bank research for own work:
Knowledge:
Low
Low
High
“Happily uninformed:”
“Frustrated uninformed:”
Do not feel the need for research and
do not have a general interest in
learning from new research.
Need to know more from research
but cannot access or finds current
research of little practical use.
Staff member’s
personal
“Independently well-informed:” “Functionally well-informed:”
familiarity with
Do not need research for own work,
Research is an important input to
the Bank’s
High but has a general interest in learning the staff member’s work and access
about development.
to relevant research is not a
research findings
problem.
How do Bank operational staff map into this matrix?
6
The survey
• GG+ staff (excl. DEC); aiming for TTLs
• Confidential, web-based, survey tool.
• Basic information on Vice-Presidential Unit (VPU), “sector”
and years of Bank service.
• A series of questions on familiarity with WB research, sources
of knowledge about that research, and the value of WB
research.
• 2,900 recipients of the survey instrument and 555 responses.
7
Response bias is a potential problem
• There was only a modest variation in response rates across
grades.
• More worrying is that the response rate varied markedly
across VPUs.
• To the extent that such differences in response rates are
correlated with perceptions of research there will be a bias for
drawing inferences about the means for the population of the
Bank’s senior staff
• Although correlations and regressions are likely to be more
robust.
• Bias corrections based on re-weighting, but can only go so far.
8
Outline:
1: Knowledge of World Bank research
2: Accessing Bank research
3: Value attached to Bank research
9
1: Knowledge of World Bank research
How familiar are the Bank’s practitioners with Bank research?
10
Knowledge about Bank research among
operational staff
“How familiar would you say you are with WB research products/
services on a scale from 1 to 10 where 1 means not familiar at all
and 10 means extremely familiar?”
.20
Familiarity with WB research
Familiarity with controls
.16
Density
• The densities are fairly
symmetric around a mode
and median of 6, and mean
of 5.74, with a standard error
of 0.10.
• Mean drops only slightly, to
5.66, when re-weighted to
allow for selective response
rates across VPUs.
.12
.08
.04
.00
0
1
2
3
4
5
6
7
8
9
10
11
11
Variation in familiarity across Bank VPUs
(Table 2)
• Less variation between regional VPUs than sectoral VPUs.
– Among the Bank’s regional VPUs, the lowest mean score for
familiarity is for EAP while the highest is for MNA.
– But few of these differences are statistically significant; only for
two VPUs—namely the HDN and PRM—are the differences
significantly different (at the 5% level) from the proportion for
the WBI.
• However, the difference between sectoral VPUs is striking
with only 36% of WBI staff rating their familiarity over 5, as
compared to 81% in HDN and 71% in PRM.
12
Correlates of VPU knowledge of Bank research:
absorptive capacity + cross-support
• VPUs with greater absorptive capacity know more about WB
research:
– High correlation across VPUs between average familiarity with
WB research and the share of staff that have “Economist” in
their job title; r=0.84.
– On comparing the proportion of PhDs (all fields) across VPUs
with mean familiarity scores the correlation coefficient is 0.66.
• DECRG “cross-support per capita” is also positively correlated
with the mean familiarity scores across VPUs; r=0.53 (1%).
• Causality unclear in all these correlations.
13
Variation across sectors
(Table 3)
• Over 80% of
staff in the PO
unit of PREM
rate themselves
as highly
familiar with WB
research, with a
mean score of 7.
• Yet less than
one third of
those in the EM
and URB had
high familiarity.
Agriculture and Rural Development (ARD)
Economic Policy (EP)
Education (EDU)
Energy and Mining (EM)
Environment (ENV)
Finance and Private Sector (FPD)
Health, Nutrition and Population (HNP)
Poverty (POV)
Public Sector Governance (PSG)
Social Development (SDV)
Social Protection (SP)
Transport (TRN)
Urban Development (URB)
Total
Average
familiarity with
research (10
point scale)
5.47
6.50
6.46
4.59
5.32
5.70
5.92
6.97
5.28
5.23
6.71
5.45
5.05
5.74
High familiarity
with research
(% above 5 on
scale
50.00
67.29
68.57
31.82
45.45
50.00
61.11
83.33
37.50
38.46
64.71
45.00
31.58
53.76
14
Two distinct sub-groups of sectors in their
familiarity with WB research
"Hard" infrastructure sectors
(ARD,EM,TRN,URB)
Mean=5.19 (0.24)
.16
"Soft" economic and HD sectors
(EP,EDU,HNP,POV,SP)
Mean=6.48 (0.14)
.12
Density
• High familiarity
with WB research
in the newer
“economic and
HD sectors”
• Low familiarity
amongst the
Bank’s traditional
“infrastructure”
sectors
.20
.08
.04
.00
0
1
2
3
4
5
6
7
8
9
10
11
The “hard sectors” accounted for 15% of
staff with high familiarity, but roughly
half of Bank lending.
15
Correlates of sectoral knowledge of Bank
research
• Mean familiarity amongst operational staff is positively
correlated with DECRG’s staffing; r=0.53.
• Also highly positively correlated with ratings of academia as a
source of research; r=0.81.
• It is not the case that the sectors and staff that know less
about the Bank’s own research tend to value external
sources (academia, external consultants) more highly.
16
Other differences in familiarity with
WB research
• Familiarity with WB research
varies with years of service and
this effect is nonlinear.
– Familiarity rises with years of
service up to 16 years and falls
after that. (20% of respondents
had more than 16 years of
service.)
– Turning point rises to 18 years
with fuller set of controls.
Familiarity with WB research
16
Years of service
with Bank
• Also lower for staff in country offices. Advances in
communication technology have not eliminated the advantages
of physical proximity.
17
How is demand for research amongst
operational staff changing?
.20
Currently rely on WB research
Expect to rely on WB reserach in future
.16
Density
1. “To what extent do you currently
rely on Bank research for your
work?”
2. “To what extent do you expect to
rely on Bank research for your
work in the next few years?”
• Strikingly bi-modal for Q1, with a
“low-usage” mode around 3 and
a high-usage mode around 7
• Lower mode largely vanishes
when one turns to Q2, about
future reliance on WB research.
.12
.08
.04
.00
0
1
2
3
4
5
6
7
8
9
10
11
Noticeably higher expected future
reliance on WB research.
18
• Those who have used WB
research in the past are more
likely to expect to use it in the
future; r=0.87 (0.1%).
• But the overall slope is less than
unity.
• This suggests that the
disparities in usage of research
can be expected to decline over
time, interpretable as
“convergence.”
Expected future reliance on WB research (with controls)
Return customers
12
10
8
6
4
2
0
-2
0
1
2
3
4
5
6
7
8
9
10
11
Past reliance of WB research (with controls)
19
12
Yet the lagging familiarity in the “hard
sectors” will continue
• For the traditional “hard sectors” (ARD, EM, TRN and URB)
their mean expected future reliance on research is 5.34 (0.28)
• As compared to 6.72 (0.16) for EP, EDU, HNP, POV and SP (and
the difference is significant; t=4.31).
• Even with the signs of convergence we have seen, the hard
sectors can be expected to persist in their relatively low
reliance on research for their work.
20
Part 2: Accessing Bank research
How do the Bank’s practitioners obtain the research they use?
21
Multiple channels, some used more than others
(Table 6)
“When obtaining research from the World Bank, how
frequently do you get the research from:…”
• The most highly-rated sources are “reports or papers,” the
Bank’s intranet, flagship reports, and the Bank’s main working
paper series.
• Seminars etc., and authored reports are close behind.
• The lowest rated source (on average) is “DECRG researcher
cross-support.”
However, staff may use source A more frequently than B
(with hence a higher score for A in the survey response) yet
B has the stronger impact on familiarity with research.
22
Impact on familiarity vs. usage
(Table 7)
• Regression of the familiarity scores on the scores for sources
of knowledge. Controls for location, sector and VPU.
• The use of DECRG researchers in cross-support now emerges
as the most important factor, with both the highest regression
coefficient and the most significant.
• Also important are informal discussions with researchers and
journal articles.
=>The direct (formal and informal) interactions with DECRG
researchers have high impacts on familiarity with WB
research, even though they score lowly in the mean ratings
for frequency of their use.
23
How well do the networks connect to research?
(Table 8)
Does your network help connect
you to Bank research?
70
Not at all
60
50
Frequency
• “To what extent does the
network that you belong to
help you navigate the World
Bank’s body of research and
researchers when you
request information?” 10
point scale from 1=“Not at
all” to 10=“Very much.”
• Mean and median of 5;
mode of 1!
40
30
Very
much
20
10
0
1
2
3
4
5
6
7
8
9
10
24
Ratings of networks in connecting to research
• Highest rated sector is
education: 70% rated
EDU’s performance as a
6 or higher, as compared
to 42% for all staff.
• Next highest in mean
score is HNP, with POV
coming third.
• The sector with the
lowest mean score is
ENV, followed by SDV.
Agriculture and Rural Development
Economic Policy
Education
Energy and Mining
Environment
Finance and Private Sector
Health, Nutrition and Population
Poverty
Public Sector Governance
Social Development
Social Protection
Transport
Urban Development
Total
Average
rating (10
point scale)
Mean
4.59
4.93
6.33
4.78
4.00
4.26
5.63
5.48
5.16
4.10
5.29
4.71
5.38
4.86
High rating
(% above 5
on scale
Mean
24.14
48.84
70.00
38.89
33.33
35.29
53.33
48.15
44.00
30.00
57.14
29.41
43.75
41.71
25
In-house or out-sourced research?
• “Which of the following statements best represents your views
about research at the Bank: (1) I would find it more valuable
and useful if the institution were to out-source more of its
research; (2) I would find it more valuable and useful if the
institution committed more resources to research within an inhouse research department.”
• 63% of respondents supported more resources for the WB’s
in-house research department.
• Those more familiar with WB research tended to be more
supportive of devoting more resources to the research
department.
26
How responsive is DEC to your needs?
(Tables 10 and 11)
.16
Responsivess of DEC researchers
Availability for cross-support
.14
.12
.10
Density
• Bi-modal, again!
• 100% of WBI staff
rated DEC as a 5+ on
responsiveness.
• As compared to 50% of
OPCS.
.08
.06
.04
.02
.00
0
1
2
3
4
5
6
7
8
9
10
27
11
Part 3: Value attached to Bank
research
How much incentive to learn about Bank research?
28
The incentive to learn depends on the value
to operational staff in their work
Density
“Please rate the overall
value of World Bank
research for your work, on
a scale of 1-10 where 1
means not valuable at all,
10 means extremely
valuable.”
• Marked bi-modality, with a
high mode around 7-8 and
a low value mode around 3.
• Mean score is 5.61 and the
median is 6. (Mean of 5.54
after correcting for
selective response.)
.16
Perceived value of WB research to own work
Perceived value with controls
.14
.12
.10
.08
.06
.04
.02
.00
0
1
2
3
4
5
6
7
8
9
10
29
11
Variation across VPUs…
(Table 12)
• The VPU with the highest value attached to WB research is
WBI, with an average score of 7, and all the WBI respondents
rated WB research highly (score of 5+).
• The next highest is PRM, with a mean score of 6.5.
• The VPU that attaches the lowest average value to WB
research is SDN, with a mean score of 5, and 45% of
respondents giving a rating of 4 or less.
• The sectoral VPUs generally put higher value on research than
the regional VPUs.
• The regional VPU with the highest mean score is (again) MNA.
30
…and sectors
(Table 13)
• Highest mean score is the poverty sector within PRM, with a
mean of 7 and for which 93% of respondents rated WB
research highly in terms of its value for their work.
• EM is the sector for which staff attach the lowest value to WB
research for their work (a mean score of 4.1, with only 41%
rating it highly), followed by TRN, URB and ARD.
• The traditional “hard infrastructure” sectors tend to put lower
average value on research for their work, suggesting lower
incentives to learn about WB research.
– ARD, EM, TRN and URB: 4.82 (0.41)
– EP, EDU, HNP, POV and SP: as compared to 6.27 (0.27)
– The difference is significant (t=2.94).
31
The matrix filled in:
Perceived value of WB research for the staff member’s work
Low
High
Staff member’s
personal
familiarity with
WB research
Low
High
“Happily uninformed:”
N=117 (22.54%)
Mean familiarity=3.35
Mean value=2.72
“Frustrated uninformed:”
N=123 (23.70%)
Mean familiarity=4.00
Mean value=6.35
“Independently well-informed:”
N=62 (11.95%)
Mean familiarity=7.26
Mean value=2.90
“Functionally well-informed:”
N=217 (41.81%)
Mean familiarity=7.58
Mean value=7.57
Note: 5+ defines “high” for the “value” variable, while it is 6+ for “familiarity.”
32
Distribution across sectors
Agriculture and Rural
Development
Economic Policy
Education
Energy and Mining
Environment
Finance and Private
Sector
Health, Nutrition and
Population
Poverty
Public Sector Governance
Social Development
Social Protection
Transport
Urban Development
Total
“Happily
uninformed”
34.21
“Frustrated
uninformed”
15.79
“Independently
well-informed”
7.89
“Functionally
well-informed”
42.11
10.28
22.86
40.91
36.36
25.00
22.43
8.57
27.27
18.18
25.00
9.35
17.14
18.18
9.09
12.50
57.94
51.43
13.64
36.36
37.50
11.11
27.78
13.89
47.22
6.67
37.50
30.77
11.76
20.00
36.84
22.54
10.00
25.00
30.77
23.53
35.00
31.58
23.70
0.00
9.38
0.00
23.53
20.00
10.53
11.95
83.33
28.13
38.46
41.18
25.00
21.05
41.81
33
The matrix filled in:
Perceived value of WB research for the staff member’s work
Low
High
Staff member’s
personal
familiarity with
WB research
Low
High
“Happily uninformed:”
N=117 (22.54%)
Mean familiarity=3.35
Mean value=2.72
“Frustrated uninformed:”
N=123 (23.70%)
Mean familiarity=4.00
Mean value=6.35
“Independently well-informed:”
N=62 (11.95%)
Mean familiarity=7.26
Mean value=2.90
“Functionally well-informed:”
N=217 (41.81%)
Mean familiarity=7.58
Mean value=7.57
Note: 5+ defines “high” for the “value” variable, while it is 6+ for “familiarity.”
34
The “frustrated uninformed”
• The “frustrated uninformed” can be thought of as a target
group for extra effort at research dissemination.
• WBI has (by far) the highest share of the “frustrated
uninformed” staff, with 65% in this category, the rest being
“functionally well-informed”—both valuing research for their
work and familiar with it, who comprise 42% of respondents.
• The MNA region has the lowest, at 13%.
• Breaking it down by sectors, it is transport staff that have the
highest share of “frustrated uninformed” group, at 35%, with
URB and SDV close behind.
• EDU has the lowest, at 9%.
35
The “happily uninformed”
• The “happily uninformed” have not simply switched to
external (non-WB) sources of research.
– They give a relatively low rating to academia as a source—a mean
score of 5.53, as compared to 6.87 for all other staff, and the
difference is significant at the 0.1% level.
– They also rate consultants lower than average, at a mean of 6.07 as
compared to 6.54 for the rest, and the difference is significant at the
5% level.
• At the same time they have an above average desire to
increase their use of the Bank’s research
– their mean difference between expected future usage and past usage
(both on the 10-point scale) is 0.80 versus 0.54 for the rest of the staff
and 0.43 for the “functionally well-informed.”
=> The “happily uninformed” are not so happy with status quo!
36
Does an incentive to learn translate into
knowledge?
• Attaching a high value to research for one’s work need not
translate into actual knowledge about research.
• That link will be broken if the Bank’s internal research
organization is unresponsive, or its products inaccessible.
• There are clearly frictions in the diffusion process for
knowledge within the Bank.
• But are they strong enough to mean that knowledge about
research does not respond to a stronger incentive to learn?
37
Does an incentive to learn translate into
knowledge? Yes
12
Familiarity with research (with controls)
• Those respondents who
put a higher value on
research for their work
are more likely to be
familiar with it.
10
8
r=0.49
6
4
2
0
0
1
2
3
4
5
6
7
8
9
10
Perceived value of research to own work (with controls)
38
How much does familiarity with research
respond to stronger incentives to learn?
(Table 16)
• Regression coefficient of familiarity on value is 0.45 (s.e.=0.04).
Going from 1 to 10 on perceived value of research for one’s
work => a mean increment of four levels for familiarity.
• Allowing for differing incentives for learning more than doubles
the share of the variance in familiarity that is explained.
• About half of the gap between the mean familiarity scores for
“hard” and “soft” sectors is accounted for by the difference in
the incentive to learn, as measured by the value attached to
research by staff working in these sectors.
• Positive is good news! But the fact that this is less than unity
suggests frictions in the knowledge response to incentives.
39
Other evidence that stronger incentives for
learning translate into knowledge
• An objective clue can be found in how the demand for paid
cross-support from the Bank’s research department varies
with answers to the value question.
• Across VPUs, the mean score for the value of WB research is
highly correlated with the per capita demand for crosssupport from DECRG; r=0.72.
• This is consistent with the view that incentives for learning in
the Bank do generate a response.
40
Why such large sectoral differences?
Both demand- and supply-side factors
have clearly played a role
41
Supply-side factors
• Staff working in the traditional sectors may well have turned
away from Bank research in part because they have found it
to be of little relevance to their needs.
• (Though they also put relatively low value on research sources
outside the Bank.)
• Today’s research priorities may well be poorly matched with
the issues faced by practitioners in these sectors.
• Making the supply of research more relevant to the needs of
development practitioners would undoubtedly help.
42
However, the supply of research is clearly
also determined by demand
• In turn, demand stems in no small measure from the extent to
which “development impact” is challenged by donors.
• Impact is often taken for granted in the hard infrastructure
sectors (though in truth the evidence is often rather weak).
• This stands in contrast to the social sectors where lending and
policy operations have had to work hard to justify themselves,
and have drawn more heavily on research to do so.
• Clearly, if the presumption of “impact” is routinely challenged
by donors and citizens then project staff will face strong
incentives for learning about impact.
• And stronger incentives for learning can be expected to yield
greater familiarity and use of research.
43