Social Assistance Pilots Program
SA Pilots Seminar
Hybrid Means Testing: An Overview
Janusz Szyrmer
March 2, 2010
Our assignment:
As part of Pilot #3:
Carry out experiments on developing new means
testing methods in Ukrainian conditions, with a focus
on the Hybrid Means Testing (HMT) methodology, as
pioneered by the World Bank
– Investigate several variants of HMT
– Test applications of the HMT methodology in five pilot
offices; the results of the experiment will not be used to
decide on the disbursement of actual social benefits to
applicants
– Use the outcomes of the experiments to draw conclusions
on the applicability of the method in Ukraine
2
In this presentation:


Social Assistance Performance
HMT Concept
• Verified Means Testing
• Proxy Means Testing
• Randomized Means Testing
• Self Selection
• Income/Non-income Testing
 HMT Procedure
 Experiments with HMT
 HMT Applications
3
SA Performance
The performance of Ukraine’s current SA system is satisfactory in terms of
targeting the recipients of SA benefits, subjected to means testing.
•
Typically verified means (income) testing is employed.
•
Also other methods are used (proxy means testing, unverified means
testing, randomized means testing, and self-selection),but somewhat
irregularly/haphazardly, often without clear rules, sometimes simply
due to limited capacities (overburdened SA office staff).
As concerns total SA transfers (all benefits and privileges, combined, in
Ukraine), the targeting performance is inferior, due to a lack of
appropriate rules and physical capacities for subjecting more
benefits/privileges to the rigors of means testing.
4
HMT Concept (1)
HMT belongs to instruments used in social assistance that:
•
Target those who need SA the most (by reducing the errors of inclusion
and exclusion), and
•
Support the efficient management of scarce budget resources
HMT is intended to:
•
Take advantage of modern means testing approaches
•
Use them broadly and flexibly
•
Design the best bundle of methods appropriate to each SA application
•
Improve efficiency of means testing methodology
5
HMT Concept (2)
HMT is a “hybrid” because it involves both direct and indirect methods,
typically:
•
Direct: Verified income testing: formal verification of incomes
•
Indirect: Proxy means testing: Incomes and/or needs assessment
based on “indirect” observable/measurable indicators that are related
to or correlated with the conditions/incomes they reflect, including
place of residence and socio-economic characteristics
Other means testing approaches/methods include:
•
Randomized means testing
•
Self-selection / self-verification
•
Also in some cases: Unverified means testing
6
Verified Means Testing
Uses formal income and asset tests to determine applicants’ eligibility for SA.
Involves checking payroll data, pensions, SA benefits, sometimes also
banking statements, vehicle documentation, etc.
In many cases the verification is automated and includes information about
tax records, unemployment data, etc.
Is effective in countries with a small shadow economy where a large share of
incomes, expenditures and wealth are formal, monetized and welldocumented. Extensive verification of information promotes transparency
and credibility (provided that it is conducted in a standard way with equal
treatment of all applicants).
Tends to be costly to implement, and both administratively and technically
difficult in CIS countries with high degrees of informality in labor markets.
It is widely used in the USA.
7
Proxy Means Testing
May involve:
Formal verification of proxies (possession of a car and other financial and
non-financial assets)
Application of econometrically estimated coefficients from a model,
typically a regression model, principal components analysis, or other
methods
Use of accounting formulas which derive “imputed income”, based on
some research results, e.g., in agricultural economics
Assigning scoring/grades to living conditions, SA needs, etc.
Use of “reliability weights” assigned to different pieces of information, in
order to generate a summary score (e.g., the Bayesian method
presented later during this session)
8
Randomized Means Testing
May be used as a cheap variant of VMT. It may deliver reasonable accuracy,
while requiring less effort in verification of household incomes. Instead of
all applicants, only a randomly selected small sample of households is
thoroughly verified.
The mathematical/statistical sciences and extensive practice provide a solid
base for the selection of random samples of households to be verified, in
accordance with rigorous procedures (the samples cannot be selected
arbitrarily by the SA staff).
If the procedure is not distorted by some irregularities, then this method may
be considered the best low-cost method which is fair, transparent and
credible. Its effectiveness may be strengthened by enforcement of some
sanctions for provision of faulty information.
9
Self Selection
Is widely used in many countries. People select themselves for SA and may
simply decide not to apply for SA, if they believe that they are not eligible.
It is facilitated by one or more of the following factors:
– Availability of accurate information to population, based on which persons who do
not qualify for SA or whose chances for getting an SA are very low do not apply
– Existence of sanctions for providing inaccurate information or social pressures which
discourage fraudulent applications
– Low value of SA (limited to a minimum in-kind or financial support) combined with
high effort (time consuming) application procedures which discourage persons with
higher incomes or those who could use the time needed for SA applications to earn
money in a different way
Self-selection is a relatively low cost method, but may be considered unfair, it
may result in a low targeting accuracy (a high error of exclusion – high
number of those who do not receive benefits despite being eligible). 10
Income/Non-income Testing
Important distinction to consider:
Income testing methods – based on “flow”(income calculation/estimation)
 Formal (documented) income
 Informal incomes, incl. imputed incomes from assets
Other methods – based on “stock”
 Assessment of
•
Living conditions
•
Consumption level
•
Access to communal services
•
Needs due to health situation, disability, etc.
In Ukraine predominantly income testing methods are used. It is advisable to design
and pilot some alternative non-income methods.
11
HMT Procedure (1)
HMT in the format developed by the World Bank is tasked with estimation of
total family income.
Total income is broken down to two categories:
(i) easy-to-verify incomes (EVI) such as official wages, pensions and
allowances and
(ii) hard-to-verify incomes (HVI) derived for instance from selfemployment, land plot use or informal activities.
Total income is calculated as a sum:
EVI + HVI
EVI is reported by an SA applicant and formally verified by SA office.
HVI is imputed by social inspectors or other SA workers on the basis of HVI
indicators, derived from an econometric model or by means of other
12
methods.
HMT Procedure (2)
SA applicants provide information on selected personal and household characteristics which
are found to be important determinants of incomes, are easy to report and verify (income
indicators).
Indicators include, for instance, demographic and social characteristics of an SA applicant and
his/her family members (such as sex, age, education, family status, etc.), work related data
(employment status and economic sector), data on possession of selected assets (e.g., a
car), data on dwelling and utilities (surface, # of rooms, equipment), on utility bills, on land
plot and livestock, as well as on location (oblast, big city, small urban area or rural area).
Each indicator has a prescribed coefficient (weight) converting it into a “portion” of imputed
income.
Applying these coefficients, in accordance with some formulas, and summing the “portions”
up generates a total amount of hard-to-verify income
Adding easy to verify income to hard to verify income generates (predicted) total family 13
income
Experiments with HMT
Extensive experiments with HMT are needed in order to design optimal rules and
procedures:
 Experiments with both income tests and non-income tests (e.g., an augmented
“Inspection Act” proposal)
 Experimental introduction of proxy means tests, as
1. Additional information source
2. An instrument for “Client profiling”
3. Extension of currently used methods (rather than their replacement),
introduced gradually in the format of biding pilots (i.e., affecting grants of SA
benefits), only in selected regions, applied only to certain categories of
applicants, such as those who:
• suffer from chronic (long-term) poverty
• do not have formal income sources or reside in regions with limited opportunities
for formal employment
14
• generate most of their incomes from agriculture or from the ownership of nonfarm assets
HMT Applications (1)
The hybrid nature of HMT enables one to combine together a few approaches in order to
establish an “optimal bundle” of methods satisfying a number of conditions, in particular:
– Political acceptability and fiscal feasibility: HMT must provide results that are feasible
in both political and fiscal terms.
– Simplicity, harmonization, low cost and administrative feasibility: HMT must be as
simple as possible and keep the cost of SA administration at a low level.
– Technical feasibility: In the economies with a large informal sector a verification limited
to formal incomes fails to assess full household incomes. An HMT method, combining
formal and informal income sources, is needed.
– Level of transparency and fairness: The assessment methods should meet clarity and
social justice requirements.
Eligibility assessment mechanisms should seek to maximize targeting accuracy at an
acceptable (low) cost and in a transparent manner while satisfying political/policy
requirements and fairness (social justice) conditions.
15
HMT Applications (2)
There are several reasons for which HMT should be considered for
applications for the needs of the Ukrainian SA system:
– International experience shows that this method may generate
satisfactory targeting results. E.g., about 85% of the benefits of HMT
programs in Chile and Mexico are received by the poorest 40% (two
lowest income quintiles) of households in those countries which is
considered a good targeting.
– There is a considerable international experience of applying HMT for SA
targeting, which may be used in Ukraine.
– It is more effective in the countries where a relatively high share of
incomes is generated by the informal sector.
16
HMT Applications (3)
HMT, relying on income proxies, possesses some limitations which must be considered in
making choices for income estimation:
– The results generated by a regression model are based on a sample which while
providing reasonable approximations remains a random set of data which is always
subject to some distortions. For various reasons people never provide fully accurate
information (due to mistakes, unwillingness to disclose information, etc.). In all
household surveys worldwide income data tend to be underreported.
– The models used data from a period during which the survey was run. For Ukraine it
was the year 2008. The data do not reflect income and price changes after 2008. Also,
the models may fail to reflect the situation of a household in which there have
occurred some recent changes of income.
– The static nature of the model requires continuous collection of data on income,
consumption and assets, with which the models must be updated. This may become
quite a challenge for SA staff required to keep recalculating the proxy income figures of
all applicants/beneficiaries.
– The income predictors generated by the HMT regression models are averages. While
many households are likely to have incomes equal or similar to these averages, in 17
some cases the actual incomes may be significantly different.
HMT Applications (4)
The HMT-type methods are promising but their introduction may require a longer
period. This can be facilitated by a number of possible improvements:
– The use of budget funds: A detailed analysis of the costs and efficiency of
the SA system is needed, which may lead to efficiency enhancements.
– National database: A systematic data collection tailored to HMT needs
should be initiated and data exchange among relevant governmental
agencies and other institutions should be strengthened.
– Analytic capacities: Specialized analytical units in MLSP and local SA offices
should be established and equipped with appropriate hardware and
software. A professional training needs to be arranged.
– Randomized means testing: Application of modern statistics methods may
be introduced, supported by appropriate guidelines and regulations from
the MLSP.
18