Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada
Sample Redesign for the FDIC’s Asset Valuation Review
David W. Chapman
Federal Deposit Insurance Corporation
Abstract
When an FDIC-insured financial institution is in danger
of failing, the FDIC has to assess the value of the
institution’s loan portfolio in a short period of time, as
part of an Asset Valuation Review (AVR). Based on the
AVR, price tags are given to various loan pools (types),
which are used to sell the bank’s loans to other
institutions, if the bank fails. Because of the large number
of loans in most portfolios, sampling must be used to
estimate the value of the loan pools. The basic design is a
stratified random sample, with strata defined by size (loan
book value). The definition of strata (including a certainty
stratum) and the derivation of the total and stratum
sample sizes have to be automated because of the limited
time available for the AVR. The current design is being
revised to improve sampling efficiency. The challenges
and features of the revised design will be discussed.
Keywords: Optimum Allocation, Stratified Sampling,
Certainty Selections
1.
Introduction
The Federal Deposit Insurance Corporation (FDIC) has
the responsibility of insuring deposits in FDIC-chartered
banks and savings associations (hereafter referred
collectively to as banks) for amounts up to statutory
limits, which is $100,000 for most standard checking and
savings accounts. In this capacity, the FDIC tracks the
financial soundness of banks so that it can be prepared to
take appropriate action when a bank is in danger of
failing. In such instances, if steps cannot be taken to
prevent a bank failure, the FDIC, as the receiver of the
banks assets, will attempt to sell the banks assets,
including the loan portfolio, to other institutions, as
quickly as possible.
To sell off the loan portfolio of a failing bank, an
important step is to assess the value of the loans, by loan
pool (type), and to put price tags on each loan pool. This
is done quickly by applying a customized computer
program to select a probability sample of loans from each
loan pool, and assessing the value of each sample loan.
From the sample valuations, an unbiased (or nearly
unbiased) estimate of the value of each loan pool (a price
tag) is derived. The sample is designed in such a way that
the value of each loan pool is estimated with a specific
level of precision (e.g., to within ± 10% of the true pool
value with 95% confidence). The process of selecting a
sample of loans, reviewing and assessing the value of
each sample laon, and pricing loan pools, is referred to as
the Asset Valuation Review (AVR).
The AVR process has to be done very quickly, often in a
few days. For over ten years the FDIC has been using a
computer program referred to as RAVEN (Risk Analysis
and Value EstimatioN) to design and select a probability
sample of a bank’s loan portfolio to be valued for the
AVR. At this time the FDIC is in the process of
converting RAVEN, which uses FoxPro software, to a
program that uses more current software. As part of this
process, the statistical methodology underlying RAVEN
is being reviewed with the intention of improving
sampling efficiency, which could save FDIC resources for
conducting AVRs. This paper describes the process of
reviewing and revising the AVR sampling methodology.
The next section provides a summary of the current
sampling methodology incorporated into the RAVEN
software. Section 3 presents recommended changes in the
AVR sampling methodology. Section 4 summarizes the
evaluation of some of the recommended changes in the
AVR sampling methodology. The last section presents
the final recommendations for revisions of the sampling
methodology for the AVR, and includes suggestions for
additional research.
2.
Current Sample Methodology Used in RAVEN
As mentioned above, RAVEN is a software package that
the FDIC uses to select a probability sample of the loans
held by a bank for the purpose of conducting an Asset
Valuation Review (AVR). RAVEN is also used to
provide estimates of the value of a bank’s loan pools,
based on sample loan valuations; but the focus of this
discussion is on the sampling methods applied in
RAVEN. The current sample design for RAVEN is a
stratified random sample. The strata are defined by loan
type (referred to as pools), and loan size, as measured by
book value (current unpaid loan balance). The same loan
pool definitions are used at all banks, though a given bank
may not contain loans in all the loan pools defined.
Prior to defining strata and deriving sample sizes, a cutoff
book value is derived for the purpose of removing the
830
Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada
smaller loans from the probability sampling process. In
particular, the cutoff is determined by sorting the loans
from smallest to largest and identifying the largest loan
such that the sum of the book values for all the loans
smaller than it do not exceed 10% of the book value of the
entire loan portfolio. The book value of the loan
identified by this process is the cutoff. All loans with
book values less than the cutoff are excluded from the
sampling process, and are valued by applying an average
recovery rate to the book value of each of these small
loans in the pool. This average recovery rate is computed
as the sum of the estimated recoveries of each of the
sample loans divided by the sum of the book values of the
sample loans.
This method of valuing loans in the bottom 10%, based
on the recovery rates of sample loans, introduces some
bias in the estimates of pool values. But the FDIC has
done some research to determine that this method is a
worthwhile bias/cost tradeoff—that the resources saved
by excluding these small loans is worth the introduction
of some bias in estimating pool values. There is an
exception to the rule for excluding loans in the bottom
10% of book values of the bank loans: If all of the loans
in a pool (or nearly all as determined by the loan
reviewer) have book values less than the cutoff value, the
entire loan pool is included in the probability sampling
process.
After removing the loans with book values less than the
cutoff, each loan pool that contains two or more
remaining loans is subdivided into two subpools, large
and small. This split is made using a procedure that
minimizes the total sample for the pool, based on meeting
a target level of precision for each of the two subpools for
estimating total book value.
To summarize this
procedure, let N represent the total number of loans in the
pool (aside from those with book values below the 10%
cutoff), and N1 and N2 (where N=N1+N2) represent the
number of loans in the large and small subpools. The
program considers 101 possible splits into large and small
subpools. For the first split, all of the loans are placed
into the small subpool (i.e., N1=0 and N2=N). The
remaining 100 splits are defined by successively
increasing N1 by 1% of N (rounded to the nearest integer).
For each of the 101 possible splits into the large and small
subpools, the required sample size is derived to estimate
the total book value (as a proxy for recovery value) for
each subpool for a specified level of precision. The level
of precision includes a specification for the relative
precision for a specified confidence level (like ± 10% at
the 95% confidence level). The calculation of the
minimum sample size needed to meet the precision target
for estimating the total book value for a subpool is based
on standard textbook formulas for deriving sample sizes,
given the subpool mean book value and variance that are
calculated from the subpool book values. The program
allows for three alternate levels of precision:
(1) High: Estimate the total book value of the subpool to
within ± 10% with 95% confidence.
(2) Medium: Estimate the total book value of the
subpool to within ± 15% with 90% confidence.
(3) Low: Estimate the total book value of the subpool to
within ± 20% with 80% confidence.
For each of the 101 splits, the required sample sizes for
the two subpools (n1 and n2) are calculated and added
together (n=n1+n2). The split for which n is smallest is
used to define the subpools. In the case of ties, the split
corresponding to the smallest value of N1 is used.
3.
Recommended Changes for the Sampling
Methods Used for the AVR
There are a number of areas for which improvements in
the sampling methodology can be made. First, since there
is a high correlation between book value and recovery
value for most loan pools, and since the distribution of
book values is highly skewed for most pools, it would
improve sampling efficiency to select the very largest
loans with certainty. There is no allowance for certainty
selections in the current RAVEN sampling methodology.
Regarding noncertainty selections, larger loans should be
selected with higher probabilities than smaller loans. This
is accomplished to some degree with the iterative method
of defining the large and small sampling strata used in
RAVEN, but the procedure is not specifically targeted to
minimizing the sample size to meet a precision target at
the pool level; instead it is focused on meeting a precision
target for each stratum (subpool). The two approaches are
not the same. For optimum (Neyman) allocation, the
stratum sample sizes are proportional to the product of the
number of loans in a stratum and the standard deviation of
the book values of the loans in the stratum. For the
methodology used in RAVEN the stratum sample sizes
are proportional variance of the loan book values in the
stratum (aside from the effect of the finite population
corrections). The total sample size and sample allocation
to strata should be focused on estimates at the pool level.
The iterative procedure used in RAVEN to define strata
will not, in general, produce optimum stratum definitions
because of the focus on meeting precision targets for the
two subpools, rather than the entire pool. Cochran (1977,
pp. 127-131) discusses a number of approximate methods
for defining optimum stratum boundaries. One approach,
due to Dalenius and Gurney (1951) is to equalize, across
all strata, the product of the proportion of loans, Wh, in a
stratum and the standard deviation of the book values in
831
Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada
the stratum, Sh. (This approach would equalize the
stratum sample sizes.) Another effective approach, due to
Ekman (1959) is to equalize the product, across strata, of
Wh and the stratum interval width, (yh - yh-1) across strata.
These are both effective methods, but require an iterative
process. Another approach described by Cochran (1977)
for defining stratum boundaries, that is used quite often in
applications, is the “Cum Sq. Root of f Rule,” due to
Dalenious and Hodges (1959). However, this approach is
defined primarily for grouped data.
size strata and stratified random sampling is
recommended. The specific methods investigated for
defining strata are discussed in the next section.
Finally, for including various levels of precision to choose
from in the sampling software (low, medium, and high), it
may be confusing to vary both the confidence level and
the amount of tolerable across the precision options. An
approach that fixes one of these precision dimensions
(i.e., the confidence level) would be easier to understand.
(4) Total Sample Size. Because of the focus in RAVEN
of meeting precision targets at the subpool level, the
resulting pool sample size will be larger than
necessary to meet the precision target for estimating
the value of the loan pool. It would be much better to
derive the required sample size based on the use of
optimum allocation to strata for estimating the total
value of the loan pool. This can be done using a
formula from Cochran (1977, p.105) for the sample
size as a function of the target variance of the
estimator (which is derived from the desired level of
precision), the stratum variances, and the number of
units in each stratum.
Based on the preceding discussion, following are several
recommended changes to the AVR sampling
methodology that should improve the efficiency of the
procedure.
(5) Precision Levels. The following three levels of
precision are recommended for choices for AVR
sampling, with the confidence level being fixed at
95%:
(1) Certainty Selections. Select the largest loans with
certainty. Since probability-proportional-to-size
(PPS) sampling is not being recommended for the
redesign for AVR sampling (as discussed later in this
section), there is no obvious way to define certainty
selections. Two criteria were tested, based on
coverage of a given percent of total book value of a
loan pool (10% and 15%). The investigation of these
two options is discussed in the next section.
High: Estimate the total book value to within ± 5%
(with 95% confidence)
(2) Number of Strata. The RAVEN software defines
two strata for each loan pool. Up to a certain point,
precision can be improved when more strata are
added. However, many of the loan pools are small,
some containing fewer than 25 loans, after removing
the loans in the bottom 10% of total book value.
Also, since ratio estimation is used to estimate the
total value of a loan pool, with book value as the
covariate, the gains from additional size stratification
may be minimal. Therefore, it is recommended that
either two or three strata be defined within each loan
pool for non-certainty selections.
(3) Stratum Definitions. A common method used to
oversample larger units (larger loans in this
applications) is probability-proportional-to-size (PPS)
sampling. However, since ratio estimation is being
used to estimate loan pool values, most of the gains
from PPS sampling can be achieved by defining size
strata (in terms of book values), using random
sampling within each size stratum, and allocating the
sample to strata based on optimum allocation
methods. Therefore, this simpler approach of using
Medium: Estimate the total book value to within
± 10% (with 95% confidence)
Low: Estimate the total book value to within ± 20%
(with 95% confidence)
4.
Investigation of the Recommended Changes in the
Sampling Methods Used for the AVR
For the first three recommended changes listed in the
previous section, various approaches were applied to
sampling bank loan portfolios for three AVRs that have
been done. The sample sizes and other features of the
revised AVR sample design were compared for various
alternatives and to those for the AVR design based on
RAVEN. In all cases, the precision target used was to
estimate the total book value (as a proxy for recovery
value) of a loan pool to within ± 10% with 95%
confidence (the medium precision level of the three listed
in the previous section.) The results from these
comparisons are discussed below.
(1) Certainty Selections. As mentioned in the previous
section, since PPS sampling is not being used, there
is no straightforward way to identify loans that are
large enough to be selected with certainty. Another
approach is for the researcher to simply observe the
size distribution of the populations units, and identify
those that appear to stand out from the rest. For AVR
sampling, it would not be prudent to ask the asset
832
Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada
review specialists to make such decisions, especially
with the severe shortage of time.
The goal is to define the two or three noncertainty
strata in such a way that the product, WhSh, is
reasonably constant across the three strata. Two
straightforward methods, that would not involve
making iterations, were tested on the AVR data. One
was to define strata my equalizing, to the extent
possible, the book value coverage in each stratum.
The other was to equalize the coverage of the square
root of book values in each stratum.
Therefore, it is recommended that the loans to be
selected with certainty are those that make up a
specific percent of the total book value of the
portfolio. Both a 10% criterion and a 15% criterion
were applied. It turned out for many loan pools that
the 10% criterion did not identify as certainty
selections one or two rather large loans that perhaps
should have been included. Therefore, the 15%
criterion is recommended.
Because of the skewness of the book value
distribution, there was concern that equalizing book
value coverage across the three strata would put too
few loans in the large stratum. Therefore, the
approach of equalizing the coverage of the square
roots of book values was investigated first. However,
equalizing the coverage of the square root of book
values placed too many assets in the large stratum,
and for a majority of pools, led to a considerably
higher sample allocation to the large stratum.
In one instance with a large loan portfolio, the 15%
criterion identified a large number of certainties. In
that case, many of the loans in the certainty group did
not particular stand out as a large loan.
Consequently, it is recommended that the number of
certainty selections for a loan pool be limited to five.
(2) Number of Strata. Revised sample designs were
prepared for most all loan pools in the available AVR
loan portfolios, defining both two and three
noncertainty strata. In most cases, the use of three
strata provided better sample efficiency (i.e., a lower
total sample size to meet the precision target) than
did the use of two strata. However, because of the
requirement of a minimum of two units per stratum,
defining three strata for small loan pools was not
beneficial. Therefore, it is recommended that only
two noncertainty strata be defined for loan pools that
contain 20 loans or less. Furthermore, for pools with
five or fewer loans, all loans should be selected with
certainty.
For all other pools, defining three
noncertainty strata is recommended.
(3) Stratum Definitoins. As stated in the previous
section, the research into constructing strata focused
on defining either two or three noncertainty strata.
The objective was to define strata for which the
optimum sample sizes for the three strata would be
nearly equal (i.e., that the product of the proportion
of loans in a stratum, Wh, and the stratum standard
deviation of book values, Sh, would be nearly the
same across the three strata.
This could be done using an iterative procedure that
derives the product, WhSh, for all possible splits into
the designated number of strata (two or three) for the
noncertainty loans in a pool. However, the gains
from defining the strata in this way may not be worth
the additional complexity associated with the
iterative procedure. Although this assessment has not
been researched, it is recommended that a more
straightforward procedure be used to define strata.
The approach of equalizing the book coverage across
the three strata worked better for defining strata for
most loan pools, and is therefore the recommended
method for defining strata. However, for both
approaches, the middle stratum often had the lowest
sample allocation because loans were often more
variable at the low end of the distribution than in the
middle.
5.
Summary of the Recommended Redesign for AVR
Sampling and Recommendations for Future
Research
Following is a summary of the features of the
recommended revised sample design for the AVR.
For each loan pool, the
(1) Certainty Selections.
largest loans that cover the top 15% of the book value
will be selected with certainty, with a limit of five
certainty selections.
(2) Number of Noncertainty Strata. For each pool
containing five or fewer loans (once the loans in the
bottom 10% of book values are removed), all of these
will be selected with certainty. For each loan pool
containing six to 20 loans, two noncertainty strata
will be defined. For pools containing more than 20
loans, three noncertainty strata will be defined.
(3) Strata Definitions. For each loan pool, noncertainty
strata will be defined using the criterion that the two
or three strata will constitute approximately equal
percentages of the book values of the noncertainty
loans in the pool. To determine stratum boundaries,
833
Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada
the noncertainty loans in a pool will be sorted from
highest to lowest, and cumulative percents of dollar
book value coverage will be computed for each loan
on the list. Stratum breaks will be made that equalize
the coverage of book values in the strata.
(4) Total Sample Size and Allocation to Strata.
For each loan pool, the total noncertainty sample
size will be derived as the minimum sample size
needed to achieve the precision target for
estimating the total book value (as a proxy for
recovery value) of the loan pool. The precision
target will be specified as either high, medium,
or low, as defined in item (5) in Section 3. The
sample size for each noncertainty stratum will be
computed using standard Neyman allocation.
There are some areas for future research, as follows:
•
Improvements of the method of defining
certainties should be investigated.
•
A number of alternate methods of defining strata
could be considered, including the use of an
iterative procedure to equalize the product of Wh
and Sh across strata.
•
The possibility of basing the total sample size
and sample allocation to strata on the ratio
estimate of recovery rates should be investigated.
There may be enough past AVR data to make
this approach feasible.
•
The level of bias introduced by leaving out the
bottom 10% of book values, and associated costs
of including these small loans in the probability
sampling process, should be reviewed.
References
Cochran, W.G. (1977). Sampling Techniques, 3rd ed.
John Wiley, New York, NY.
Dalenius, T. and Gurney, M. (1951). “The Problem of
Optimum Stratification.” II.Skand. Akt., 34, 133-148.
Dalenius, T. and Hodges, J. L. (1959). “Minimum
Variance Stratification.” JASA, 54, pp. 88-101.
Ekman, G. (1959). “An Approximation Useful in
Univariate Stratification.” Annals of Mathematical
Statistics, 30, pp. 219-229.
834