Exercise 14-3: Power

Exercise 14-3: Power
PROJECT BACKGROUND
Your team is evaluating the impact of a local governance program that provides training
and outreach at the community level on participatory budgeting. Your primary outcome
of interest is increased citizen participation in the budgeting and monitoring process,
particularly from marginalized groups. The evaluation team has settled on a clustered
randomized evaluation where communities are randomly assigned to the treatment or
control group. During the planning phase, the IE team conducts revised power
calculations based on data from a similar project implemented recently in the country by
another donor.
TEAM TASKS
Consider the following scenarios and questions related to the power analysis:
1. In doing power analysis, how would you set the relative level of significance for
these cases (i.e. would you require a higher level of significance for (a) or (b))?
a. Pilot project will be the basis for large scale funding allocation
b. One-off project where we think we have two effective approaches but want
to get a sense of whether one is more effective than the other.
2. Your IE contractor recommends using a minimum detectable effect size of 0.6
SD. You notice that this is the same effect size that was identified in the similar
program funded by the World Bank. Do you agree with this?
a. If so, why?
b. If not, what would you recommend (or how would you identify a different
effect size)?
3. No secondary data exists, and data collection is expected to be expensive. The
budget only allows for a sample size sufficient to reach a power of 30%. The
team feels there is no flexibility in the assumptions in the calculation. Do you
recommend continuing with the evaluation?
4. Since the evaluation uses a clustered design, you are concerned about the intracluster correlation (icc). You know that the final sample size, due to budget
limitations, will be 2,000 (equally split between treatment and control). The
1
program will operate in 150 communities. You have two choices for how to
structure your sample:
a. Collecting data from 100 individuals in each of 20 communities, 10
treatment and 10 control, for a total sample size of 2,000
b. Collecting data from 10 individuals in each of 200 communities, 100
treatment and 100 control, for a total sample size of 2,000
4.1 To minimize the effect of icc on power, which scenario would you
recommend?
4.2 Which scenario do you think would be the cheapest to implement?
2