AN EXAMPLE OF COOPERATION
& SOME WIDER ISSUES
Ian Plewis
(Bedford Group, Institute of Education) &
Stephen Morris
(Social Research Division, Department for Work &
Pensions)
OVERVIEW
• Government research identifies important
methodological issues
• These issues often/sometimes ignored or
assumed not to be a problem
– exception ONS work on methods & quality
• Where methods research takes place no
government forum for exchange exists
• Child outcomes work (IoE/DWP) – a methods
project - to illustrate the problem
• We suggest why methods often not addressed &
thoughts on potential solution
THE RESEARCH
• A feasibility study
• Are adult outcomes for individuals (parents)
linked to (or caused) by changes in childhood
circumstances?
• Effects of change in household income
(household deprivation & parental employment)
in childhood on individual’s adult outcomes
• Pathways through which changes take effect?
POLICY CONTEXT
• Government child poverty targets – end child
poverty by 2020.
• This research informs development of policies to
meet targets
• Marginal £ - what be the most effective way of
addressing child poverty
– Income transfers?
– Services?
• Our focus on changes in income thus whether
transfers are likely to be effective
WHY NEW RESEARCH?
• Not much UK research looking at change (Our
interest is in establishing causality)
• Tends to focus on short-range educational
outcomes
• Focus on single data sources
• Most useful sources – birth cohorts – have
limitations
• Can data sources be combined to better address
questions of relevance
– What assumptions do we have to make?
– What statistical models could be used?
THE PROBLEM
• Most quantitative social research has to deal
with methodological issues.
• Some of these issues are ‘cutting edge’ and
need to be addressed within a research council
project.
• Some are ‘standard’.
• But many fall between these two extremes and
tend to get ignored in Government funded
projects
MEASUREMENT ERROR
• Suppose we want to estimate a regression
model with a measure of income as one of
possibly several explanatory variables.
• We know that measures of income are
unreliable and we also know that this
unreliability, if ignored, can lead to biased
estimates of coefficients of interest.
• We should correct for measurement error
because not to do so could lead to misleading
inferences for policy.
• But such corrections are not standard and the
problem is often swept under the carpet. .
NON-RESPONSE
• Unit non-response and attrition are features of
all longitudinal datasets.
• Certain kinds of cases tend to be lost – at the
outset and over time.
• Ideally, we should carry out sensitivity analyses
of some kind to establish whether losses from
the sample are likely materially to affect policy
conclusions.
• But these analyses take time and resources
INCOME MISSING
• Data are often missing for income even when
other variables are measured.
• And some surveys do not collect income data.
• Imputation is a way round some of these
difficulties.
• But there are a number of imputation methods –
how does their application affect our
conclusions?
IMPLICATIONS FOR POLICY
• Measurement error/non-response = biased
estimates
• Biased estimates can lead to bad policy
decisions
• Example (hypothetical):
– Under-estimate of effect of change in income on child
outcomes
– Assume income transfers less effective
– Direct policy toward services
– Sub-optimal policy response
WHY MIGHT THIS HAPPEN?
• No cross-government forum for exchange of
information on methods research
• Within spending departments not always easy to
make the case for methods research
– Benefits of methods research diffuse
– Not priority for policy customers
• Researchers don’t always appreciate
implications
• Time – complex problems take time to solve
WHAT COULD BE DONE?
• Establish a GCSRO/NCRM methods group
(representatives: depts., ONS, ESRC & NCRM)
• Forum for raising methods related issues with policy
focus – disseminate existing work from ONS, depts. &
academics/researchers
• Identify gaps and commission – requires funding
– Policy relevance
– Wider application
• Dissemination – publications, seminars and training
courses
• CASE studentships
CONTACT DETAILS
Ian Plewis – Institute for Education
(T) 020 7612 6238
(E) [email protected]
Stephen Morris – Department for Work & Pensions
(T) 020 7962 8193
(E) [email protected]