Auto-evaluation of FAO
outlook studies work programme
Adrian Whiteman
FAO, Rome
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Introduction
• FAO has to auto-evaluate each work
programme element every 6 years
• Surveyed OS users and experts Oct-Dec 2004
• Present the results of the evaluation
- achievements
- guidance from WP
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Auto-evaluation survey
• Users (from various FAO databases) –
internet, e-mail and paper questionnaire
• Experts and policymakers – telephone
interviews
• FAO staff – telephone and face-to-face
interviews
• Citation analysis
• Website statistics
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
User survey - coverage
Type of questionnaire
People who
requested OS
Other
people
Total
1. Distributed to mailing lists provided by FAO
863
173
1036
2. Number of invalid emails
325
46
371
3. Number of valid emails (1-2)
538
127
665
4. Number of tracked responses from email lists
176
54
230
5. Percentage of tracked responses (4 / 3)
33%
43%
35%
X
55
55
176
109
285
6. Anonymous responses
Total responses
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
User survey - coverage
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
User survey - coverage
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Importance of outlook studies
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Knowledge of outlook studies
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Knowledge of outlook studies
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Quality of outlook studies
Overall how do you rate the quality of FAO’s forestry sector outlook studies?
Good
64%
Very good
25%
Don’t know
8%
Poor
Very poor
3%
0%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Response percent
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
90%
100%
Quality and comparability
In comparison with the work of other organisations which of the following
statements would you agree with the most?
FAO’s forestry sector outlook studies complement the
information I can get from other sources
40%
FAO’s forestry sector outlook studies are better than
other sources of similar information
18%
FAO's forestry sector outlook studies are comparable
with other sources of similar information
17%
FAO’s forestry sector outlook studies are the only source
of this information available to me
12%
Don’t know
12%
FAO’s forestry sector outlook studies are worse than
other sources of similar information
0%
0%
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Response percent
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Other sources of outlook studies
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Outcome: improving knowledge
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Priority for future commitment
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Suggested focus in the future
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Survey of experts and policy makers
• 83 people approached: experts; opinion
formers; contributors
• 70 people contacted
• One-third non-users; one-fifth only slight users
(not getting to opinion formers – especially in
government)
• Some significant users: CIRAD; METLA; Forest
Trends; CIFOR; World Bank; WWF; African
Development Bank; Conservation International
• 35 people interviewed in detail
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Strengths and competitive advantages
• Unrivalled database
• Credible, unbiased, transparent and free
• Long-term and multi-country analysis, not
available elsewhere
• Have the authority of FAO’s backing, especially
in broader agriculture and land-use context
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Shortcomings and limitations
•
•
•
•
•
•
•
•
Not well advertised and weak follow-up
Audience is unclear – need more focus
Too big and too long/detailed for policymakers
Not enough country-specific follow-up
(especially in Africa)
Presentation is sometimes a problem
Overly cautious
Short shelf-life
(Experts/contributors) – data quality is an issue,
sometimes a heavy demand on contributors
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Citations (total 105)
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Citations – use of outlook studies
• PUBLIC SECTOR: used to stimulate debate,
explain policy and raise awareness of the sector
• PRIVATE SECTOR: used to stimulate investment
• ACADEMIC: used for research planning
• In particular, outlook studies are used to debate
environmental issues, then economic and social
issues in the forestry sector
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Survey of (37) FAO staff
• Mixed response
• Not used extensively, mainly used for
information rather than analysis
• Unclear focus
• Poor marketing
• Little follow-up in countries
• Short shelf-life, need regular updates
• Should work more holistically
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva
Comments, questions and guidance
please! Especially:
• Audience: national policy; international policy;
investment – need better focus?
• Large studies every 10 years or shorter more
regular studies?
• Process – top down or greater country
capacity building?
• Marketing – getting the product onto desks!
• Other issues (e.g. future developments)?
Joint UNECE/FAO Working Party on Forest Economics and Statistics,
22-24 March 2005, Geneva