MPA-T-4
BEFORE THE
POSTAL RATE COMMISSION
WASHINGTON DC 20268-0001
POSTAL RATE AND FEE CHANGES,
2000
i
)
Docket
DIRECT TESTIMONY
OF
KEITH HAY
ON BEHALF OF
MAGAZINE PUBLISHERS OF AMERICA, INC.
ADVO, INC.
ALLIANCE OF NONPROFIT MAILERS
AMERICAN BUSINESS MEDIA
ASSOCIATION FOR POSTAL COMMERCE
ASSOCIATION OF AMERICAN PUBLISHERS
COALITION OF RELIGIOUS PRESS ASSOCIATIONS
DIRECT MARKETING ASSOCIATION, INC.
DOW JONES & COMPANY, INC.
MAIL ORDER ASSOCIATION OF AMERICA
THE MCGRAW-HILL COMPANIES, INC.
NATIONAL NEWSPAPER ASSOCIATION
PARCEL SHIPPERS ASSOCIATION
and
TIME WARNER INC.
No. R2000-1
Sketch . . .. . .. . . . . . . .. . .. . . .. .. . .. . . .
2
L
Autobioqraohic
!L.
Purpose and Scope of Testimony . . .. .. .. .. . .. .. . .. . .. .. . .. . . .. . .. . .. . . .. . .. . .. . . . . . . .. . . . 4
L
Value Of Research in Decision Makinq . . .. . .. . .. . .. . .. .. . .. . . . . .. .. . . . . .. .. . . . . . . . . . . 5
a.
Scientific Method in Statistical Studies . .. . .. . .. . .. .. . . . . .. . .. . .. . . .. . .. . .. . .. . .. . .. . . 5
!L
Desianinq the Sample Frame .. . .. . .. . . . . . . . . . . . . . .. .. . . .. . .. .. . . .. . .. . .. . .. . .. .. . 7
i
Randomness
L
Samole Size and Cost/Confidence
nl.
Stratified Sampling . . . . .. . .. . .. . .. .. .. . .. . .. . .. . .. . .. .. .. . . . . .. .. . . . . .. .. . .. . . . . . . . 10
L
Cluster Sampling .. . .. . .. . .. . .. .. .. .. . .. . .. . .. . .. . .. . .. . .. .. . .. . . .. . .. . .. . . .. . .. . .. . .. . .. . .. . 10
L
Multiole and Seouential
)&
Choice of Samole Methods . .. . .. . .. . . . . . . . . . . . .. . .. . .. . . .. . .. . .. . . .. . .. .. . . . .. . . 1 1
a
Trial Samole Testino and Lessons Learned . .. .. . .. . . . . .. .. . . . . .. . .. . .. . .. . .. . 12
Iv.
Data Collection
i
versus system
7
Considerations
8
Samolinq
10
Issues .. . .. . .. . .. .. .. . .. . .. . .. . . . . . . . . . .. . .. .. . .. . . . . .. .. . . .. .. . .. . .. . .. . .. . 12
Questionnaire
desion - “Answers to Questions”
v. “Questions
made to
fit the answers ” . . .. . .. .. .. . .. . .. . .. . . .. . . . .. . .. . .. . .. . . . . .. . .. .. . . .. . .. . .. . . .. . .. .. .. . .. 12
L
Enoineerino
y.
Enumeration
k
Trainino manuals and loo-books . .. .. . .. . .. . .. . .. .. . .. . . .. .. . .. . . .. . .. . .. . .. . .. . .. . . 15
!L
Trainino the trainers
a
Trainina the enumerators
m
Decision Rules on Data Acceptance
ylll&
Data Qualitv Maintenance
Iv.
estimates versus cost estimates
13
methods . .. . .. . . . . .. . .. . .. .. .. .. .. .. . .. . . .. . .. . .. . . . . .. . .. . .. . . .. . .. . .. . 14
15
. .. . .. . .. . .. . . . . .. . .. . . . . . . .. . .. . .. . . . . .. .. . . . . .. . .. .. .. . 16
. .. . .. . .. . .. . . . . .. . .. . .. . . .. .. . . . . .. . .. . .. .. . . 16
.. . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. . .. . . . . .. .. . . . . .. . .. . .. . . 16
Are any data better than no data?
17
L
The need for new USPS cost data. .. . .. . .. . .. . .. . .. . . .. . .. . .. . . . . .. .. . .. . . .. . .. . .. . . 17
&
Quality Data for Quality Decisions . .. . .. . .. . .. . .. . .. . . .. . .. .. . . . . .. .. . . . . .. . .. . .. . . 18
m
What the Data Qualitv Study said about Letter Carrier Costs . . .. . .. . .. . . 18
k
Is the Enqineerino
Studv data better than no data?
19
1
2
I.
Autobiographic
Sketch
My name is Keith Hay, I am Professor of Economics
3
4
in Ottawa,
Canada.
5
company specializing
I was educated
6
I am also the President
in economic
of Econolynx
at the University
of Toronto,
a
States.
9
Foundation
Fellow.
10
Information
Research
11
of Southern
California;
12
Alberta (Japan Foundation),
in Canada;
I was a U.K. State Scholar,
I am also
International
in the United Kingdom;
and Brown
University,
a Ford Foundation
a Fellow
of the
York University,
Foundation
in Ontario Canada;
century,
some
15
organizations
16
American
17
Development
18
international
19
assistant to Simon Reisman - - the “father” of the Canada-US
20
Canada-US
21
formulating
22
the proposed
23
gains and losses.
24
often acting
I have undertaken
research
as: the World
as an international
Bank, the Asian
the Organization
trading
Free Trade Agreement
and
States,
banks.
for such
Bank, the Inter-
the Canadian
of American
companies
two hundred
consultant
Development
Bank, the Bank of Canada,
corporations,
at the University
and the University of
14
Agency,
for Advanced
in Canada.
Over the last quarter
Development
in the United
Fellow and a Killam
in Japan. I have been “Visiting Professor”
assignments,
Ltd., a
research.
at the University of Southampton,
7
13
at Carleton University
International
and numerous
I was
executive
Auto Pact and the
- - during the period when
Canada
was
its modern free trade policies. Most recently, I have been working on
Canada-Japan
I have worked
Free Trade Agreement,
for Canada
the development
Post on a number
25
significantly,
26
competitor
27
as an adviser to several Canadian
26
the CEO of a publicly quoted software company.
29
Canada, and I live in Ottawa, Canada.
service standards,
and maintenance
marketing
the potential
of assignments,
of a large database
incentives
high-technology
2
assessing
and customer
companies
most
of parcel
rates. I serve
and I have been
I am a citizen of both Britain and
1
Perhaps
the most significant
employed
experience
I bring to these proceedings
me as the technical
is
2
the fact that A.T. Kearney
editor on the Data
3
Quality Study. I was tasked with reading all the component
studies compiled
4
the various experts to ensure that they read well individually,
and that collectively
5
they had some cohesion.
6
various data quality issues at length. I believe this gives me an excellent
7
into the subject of “Data Quality and Rate Making.”
As such I met often with the authors and discussed
3
by
the
insight
1
IL
Purpose
and Scope of Testimony
2
Data quality is fundamental
3
to sound decision
making based on sample
4
statistics. Good decisions must therefore rely on good data. Postal rate making in
5
the United States is viewed from the outside as setting “best practices”
6
world, by adhering
7
application.
8
parties to replicate statistical
9
the high standards
to the highest standards
Accountability,
desired
transparency,
of scientific
methodology
for the
method and statistical
and the ability of third
methods and sample results are the key-stones
by the Postal Rate Commission,
10
Postal Service and its end-user stakeholders.
11
The cost of mistaken decisions
the United States
based upon inappropriate
be severe for the stakeholders
of
cost estimates
12
could potentially
13
rate-makers.
14
integrity are lowered,
15
prove difficult to enforce in the future. While the desire for a quick answer or fix
16
may be understandable,k
17
to the shareholders
18
derived from applying non-random
Moreover,
and for the credibility
there is no going back; once the standards
the floodgates
will open and science-based
for research
rationality
and American
to consistent
consumers
than any benefit of a quick answer
and judgmental
20
adherence
21
variable
22
about consistency
23
holders to buy into results, then the interpretation
24
treated very warily.
25
Engineering
26
exhibit many of the afore-mentioned
27
In and of themselves,
decision
statistical procedures.
manuals
for enumerators
of data collection, and only expost
28
to play in determining
29
the data acquisition
limits, weak
random sampling, no statistical cost study questionnaire,
rules, no training
Recent work by Mr. Raymond
Standards
will
the risk of making a mistaken decision is much greater
When there is no study design, a lack of pre-set confidence
19
of the US
Study to produce
or great concern
facto attempts to get stake-
of the arising results must be
and Mr. Baron reworking
inputs for cost-estimates
appears
an
to
shortcomings.
Engineering
Standards
studies have important roles
time and motion aspects of route performance.
However,
methods applied in ES research are quite different and often
A
1
inappropriate
2
applied “any data” (arising from the ES study) may be worse than “no data” (from
3
statistical
4
be justified. The budget costs - and delay -- of undertaking
5
well designed,
6
immediate
7
accuracy, data quality and avoidance
10
making;
purposes.
This is a situation
cost studies) and the compounding
statistical
of decision
study, as suggested
in which
a scientifically
sound,
by the Data Quality Study, in the
in sample
of rate making errors.
I review some issues of statistical
look at concerns
wrongly
making errors cannot
future are dwarfed by the likely value of the improvements
In my testimony,
8
9
for ratemaking
about data collection
research
methodology,
in decision
and discuss
the
question “is any data better than no data?”
11
12
Ill.
Value Of Research
in Decision
Making
a.
Scientific Method in Statistical Studies
13
14
15
16
As is well known, there is a long history of the use of scientific method in
17
survey
18
century to the issue of obtaining estimates
19
based
20
research project requires careful planning, which involves:
research.
Probability
upon random
sampling
theory
has been ably applied
of the parameters
of that population.
for almost
a
of a population
The structuring
of the
21
22
consulting
early with clients, end-users
23
be affected by study outcomes (stakeholders);
24
reviewing previous studies/literature;
25
determining
26
fulfilled;
27
adopting the appropriate
28
establishing
a set of questions
acceptable
and decision-makers
to be answered
or objectives
null hypotheses;
confidence
5
limits for the desired results;
likely to
to be
1
selecting a random sample frame and method - - for instance stratified
2
sampling, cluster sampling - - and/or multiple applications
3
developing
4
users
5
outcomes;
6
making sure that the answers fit the questions - - not that the questions
7
fit the answers;
8
running a pilot study to refine procedures;
9
incorporating
lessons learned from the pilot study;
10
establishing
decision
11
issues before they arise;
12
devising manuals to guide enumerators
13
ensuring
14
strata or clusters by means of training, handbooks
15
training
16
quality control;
17
recognizing
18
subject to analysis and measuring
19
handling
20
integrity of the overall methodology;
21
presenting
22
appropriate.
a questionnaire
and those
who
consistent
the trainers
of these;
with expert advice from the client, end-
will base
their
decisions
on the
rules to deal with sampling
methods
and data quality
and analysts;
of data collection
and emphasizing
the importance
of moments
continuity
across
results
with
the sample
and logbooks;
and consistency
of demarcation
in
in activities
them with a keen eye to precision;
the data with care with a view to preserving
the
research
the scientific
and
suitable
disclaimers
as
and
when
23
24
All of this, of course, to be achieved on a research budget which is always - - by
25
definition - - too tight, and within a timeframe that is inevitably too short!
26
are not easy tasks.
27
research
28
confidence,
29
researchers,
approach,
But in general, the stricter the adherence
These
to the pre-designed
the more likely are the results to be usable
with known
while the quality of the resulting data will more likely be acceptable
clients and end-users
as a whole.
6
to
Research
1
design, sample randomness,
accuracy
2
transparency
3
another
4
with these same data, could the original results be replicated’?
5
6
7
a
9
10
b.
are fundamental
enumeration
researcher
and overall
to the ability to positively answer the question:
independently
undertook
to answer
“if
these same questions
Designing the Sample Frame
i. Randomness versus system
Statistical
analysis
is used
11
population
under examination.
12
purpose in order to know more about the population
as a whole.
13
because
of the population,
14
successive
15
to appropriate
16
mind the issues associated
of ever-present
accurate
A sample
variation
and reasonable
task.
is selected
about
conclusions
about the population
for this
Difficulties arise
such that
is to come
while bearing in
with sampling variation.
must cope with two key requirements
analytical
The first is to design
19
sampling
20
the sample results to draw correct inferences
21
most difficult
22
Inferences
23
competently.
24
attributing to the population
so that it is representative
to achieve
are unlikely
the parent
and observed
The task of the researcher
ia
a sampling
the second
and undertake
objective
the
and the second is to use
about the population.
Clearly, it is
if the first is not well done.
to be accurate
unless
the sampling
procedure
Therefore,
in carrying out the
frame
of the population,
the sample
has been taken
must be acceptable
before
results arising from an analysis of the sample.
In general, for samples to contain worthwhile
25
inferences
among elements
samples are usually different.
The researcher
17
to make
and reliable information
about
26
the population,
each unit of the sample must be selected at random, requiring that
27
each element
of the population
28
sample.
29
or interested third parties and they exercise their own choices, then the probability
If selection
has a known
is left to the judgement
7
probability
of the researcher,
of appearing
in the
his/her associates
1
surrounding
these selections
2
statistical procedures
unknown
and the application
of standard
is confounded.’
A common procedure
3
becomes
for ensuring
randomness
in a sample is to leave its
4
drawing to a mechanical
5
the control of the research
6
applies when samples are stratified and/or clustered
(as noted below).
7
can admit that pure randomness
in research
a
fundamental
aim
9
mathematical
model upon which the preponderance
10
11
of
process, such as a random number generator,
team and interested
is rarely attained
statistical
The closer the researcher
accurate will be the inferences
parties.
research
This argument
methodology,
can approximate
beyond
While we
practice,
and
also
it is a
invokes
the
of statistical theory relies.
randomness,
the more nearly
drawn from the research study.
12
13
ii.
Sample Size and Cost/Confidence Considerations
14
Given
15
that
procedures
are
in sample selection,
in place
to achieve
16
randomness
17
size? If the sample size is too small - it may be too inaccurate
ia
large a sample may require the expenditure
19
little extra information
beyond
20
useful
At issue is a determination
21
researcher
22
decision on an acceptable
23
of the results
24
magnitudes
sample
size.
degree
of
a key issue is how large must be the sample
to be reliable.
of too many resources
what could be obtained
and his stakeholders
a high
Too
while adding
from some smaller
yet
of how large an error the
can live with in the estimate.
Moreover,
the
error also must take into account the uses to be made
and the potential
cost and revenue
consequences
of error - - for the client and other end-users
of different
who may ultimately
be
’ Non-probabilistic
sampling procedures, such as quota sampling and convenience sampling,
represent judgement samples, since they involve the selection of items in a sample on the basis
of opinion, not randomness. When the population is small, or time/money will not allow collection
of a random sample, or the study is strictly exploratory, then a judgement sample may be
justified, but the statistical implications of abandoning random sample selection should be well
understood, should be clearly flagged and should be expected to attract comment.
a
1
affected by the use of these results.
2
validity of inferences to be drawn from the data.
3
This goes to the issue of data quality and the
Put another way, the researcher
should set up an allowable
4
of confidence
limits, before designing
5
sample size - - overall, by strata and/or by cluster.
6
Once
a decision
7
stakeholders
8
will exceed the allowable
9
95% confidence
of this
the sample
type
is made
frame and deciding
by the
researcher
on the
and
his
- - say that they are only willing to take a 5% chance that any error
error in the sample mean - - then they have selected a
limit for their study.
With this decision in hand, there are then a
10
number of ways to estimate what is an acceptable
11
undertaking.
12
or related populations),
13
sampling,
14
needed of both the relative costs per unit of collection
15
the strata and/or cluster under observation;
16
size indications
17
error, in terms
sample size for the research
These require bearing in mind prior information
results of pilot studies,
and budget
Essentially,
constraints.
that are acceptably
statistical
some
(from earlier studies
methods
advance
for complex
estimates
and expected
variance
are
in
rough estimates will often give sample
close to an optimum allocation.
Simple random sampling of a large population
it might prove very costly.
may be difficult to achieve,
18
not least because
19
employed
recognizing
that they
20
discussion
and dispute.
Among the methods that may be employed
will also
More practical
be more
procedures
restrictive
and
may be
open
to
are:
21
22
23
24
25
26
l :* Systematic
sampling - choosing
a random starting point and selecting
every Kth element to be an item in the sample:
l :* Stratified sampling -dividing
a population
into homogeneous
groups or
classes as strata. Each stratum is then randomly sampled;
l :* Cluster
sampling - where the parent population
is sub-divided
into
27
groups so as to design an efficient sample.
These clusters ideally have
28
the
population
29
randomly sampled.
same
characteristics
as the
9
parent
and
are then
1
2
iii. Stratified Sampling
3
4
The best method of selecting strata is to find groups with a large variability
5
between strata, but only a small variability within the strata. Choice among and
6
within these groups may then be based on a random selection method.
7
A proportional stratified sampling plan would use items from each stratum
8
in proportion
to the size of that stratum, to ensure that each stratum in the sample
9
is weighted
by the number
10
population.
11
known that a particular
12
a maximum
13
weighting of such results should reflect the proportionality
14
the sample strata.
of elements
it contains,
relative
to the parent
A disproportionate stratified sample may be an efficient device, if it is
amount
stratum contains a high degree of variability that will yield
of information
for a given amount
of research
effort.
or dis-proportionality
The
of
15
16
iv. Cluster Sampling
17
18
The objective
here
is to obtain
observations
such
that there
is Me
19
variability between clusters, but a high degree of variability - - representative of
20
the parent population - - within each cluster. If each cluster is assumed to be
21
representative
22
can be estimated
23
within this cluster.
24
and efficient.
of the parent population,
then the characteristics
of the population
by randomly picking a cluster and randomly sampling
Two-stage
elements
random sampling within a cluster is often effective
25
26
v. Multiple and Sequential Sampling
27
28
29
When budget constraints
impact sample design, it is often useful to frame a
pilot study wherein only a small number of items are used to represent
IO
the parent
1
population.
2
multiple-stage
If high variance
4
when fewer
5
cluster.
then it may be valuable
sampling, especially when the parent population
The advantage
3
is uncovered,
of sequential
to undertake
is large.
sampling is reflected in the savings that result
items than usual must be observed,
say from a cluster
within a
6
7
vi. Choice of Sample Methods
8
9
Selection from among several types of random sampling plans depends on
10
the researchers
11
previous research);
12
achieve efficient and confident
13
address include:
14
1) What is the most cost effective method to collect samples
15
16
17
18
19
20
prior knowledge
of the parent
population
(and the results of
namely the likely validity of stratified and cluster sampling to
parameter
that the samples are representative
2) How reliable are the inferences
estimates
of the population.
that best ensures
of the parent population?
and conclusions
about the parent population
likely to be drawn from sample information?
3) What are the best ways of describing
overstating
sample information
usefully while not-
the predictive power of the results?
It is the decisions resulting from incorrect inferences that can be costly, not
21
the incorrect inferences themselves. Thus, there is a requirement
22
client and stakeholders
23
making an incorrect decision, or error.
24
25
Issues to
that the sampling methods employed
on behalf of the
minimize the cost of
At the end of the day, a primary objective of sample design is to balance
the potential costs of making an error against the costs of undertaking
26
27
28
29
11
sampling.
vii. Trial Sample Testing and Lessons Learned
1
2
3
Pilot studies, which use the overall research design, sampling methodology
4
and questionnaire
5
clients and stakeholders
often refines the issues, tightens the project focus, and
6
sharpens
tools.
7
acceptance
8
statistics
9
overall sample size to achieve best value for money within the confidence
set-up, yield valuable
the statistical
of the research
gleaned
insights.
It also ensures
end results.
Discussion
of pilot results with
a higher degree
Moreover,
information
from the pilot can be very helpful in deciding
10
acceptable
11
the decision
12
and analytical
13
reveal potential pitfalls in avoiding bias in the final results.
14
explain the choice or and rationale for one of these methods
15
provide the target confidence
16
study
17
unnecessary,
is
to the clients and stakeholders.
- rules adopted
methods.
exploratory,
concerning
In summary,
or
of stakeholder
on the optimal
It also allows the researcher
limits
to test
data quality, data inclusion/exclusion,
results from a pilot or trial sample usually
Studies which neither
of sampling,
limits should be viewed with concern.
its
on sample
conclusions
should these rigorous standards
regarding
the
parent
nor
Only if the
population
be relaxed,
18
19
IV.
Data Collection
Issues
20
21
22
23
i. Questionnaire design - “Answers to Questions” v. “Questions made to fit
the answers. n
24
It almost goes without saying that questionnaire
25
to achieving useful results.
26
designed
27
you plenty of examples
28
distorted
29
unexpected
30
wording
questions
should be consulted.
elicit difficult to interpret answers.
concerning
do you want
questionnaire
will usually
Clients and stakeholders
questions
the answers
responses
remove
design is very important
Any Canadian
about “Quebec
to be?!
Badly
will give
Separation”
Pilot surveys
usually
due to a poorly framed interrogatory.
potential
response
12
biases.
Perhaps
- how
reveal
Re-
the most
1
alarming
2
conducted
3
answers”
4
survey questions,
how could they exercise
5
being
ex.post, or recognize
6
endemic?
7
of errors in data collection,
8
relevant
questionnaire
9
designers
and the enumerators
approach
is to design
(for another
in some fashion.
measured
any data
its objectives
12
different “unspecified”
13
observation
14
categories.
15
appears
16
reasons earlier discussed,
17
sample or the parameter
were
unknown
All answers
to the
questionnaire
21
standards.
research
answers in his cost
recorded
were based on a
into a new set of questions
is doing the complete
fitting the
- the six cost
and divination.
exercise
It
backwards. For
it is not possible to offer any level of confidence
in the
estimates arising therefrom.
A typical cost study questionnaire
20
or
and decision rules, since the
How well he has done this is a matter of conjecture
and
random
leaves in tatters all the issues
set of criteria. This is an instance of a researcher
tallies, i.e. “the answers”
to be observed
-
until affer the data had been collected.
study were posed to enumerators.’
19
over what was
deficiencies
none of the questions that Mr. Raymond
as if the researcher
fit the
did not know these post-
any quality control
data exclusion/inclusion
and
has been
to make “the questions
Since the enumerators
11
18
and attempt
This “cart before the horse” procedure
For example,
10
purpose)
after the survey
the questions
the
design would clearly specify the activity
points at which if begins and ends. No such
exists for these data nor are there any relevant
observational
22
23
24
25
ii. Engineering estimates versus cost estimates
There
is a remarkable
difference
between
quantifying
the number
of
26
sufficient time and motion segments for an engineering study of time use, versus
27
quantifying
the appropriate
number
of routes,
by route
type
to develop
’ See L. Raymond, Direct Testimony on behalf of the United States Postal Service before the Postal Rate
Commission, Washington, DC. 20268~Oool, Docket No. R2000-1, USPS-T-13 and his Library Reference
to USPS-LR-1-163, Engineered Standards Database.
13
a
1
statistically
2
decisions.
valid
Industrial
3
sample
Engineers
4
pieces
5
estimates.
6
errors are not cumulative,
7
inherent
8
individual
9
statistical cost studies.
10
of work, which
In
for purposes
of cost
(I.E.) use sampling
are not necessarily
The I.E. advantage,
techniques
so as estimates
are added
addition,
I.E.
estimates
or low productivity.
often
used in cost
together,
of accuracy
distinct
is that the individual
This enables
exclude
rate-making
to measure
from a statistical perspective,
to a lower degree
and
the same as those
bias exists, the total error reduces.
work elements
estimation
provided
no
them to measure
than is called for in
any
time
11
inefficiencies
12
is essential that the sampling for cost studies be constructed
13
bias from these factors. The various aspects and distinct elements
14
cannot be merged together - - as in I.E. - - without recognizing
15
significant
measure
for
As cost estimates capture these two elements it
so as to avoid any
of load time
that there will be
losses in accuracy and variability for cost estimation
purposes.
16
17
iii. Enumeration methods
18
19
The method by which Mr. Raymond conducted
20
the Engineering
Standards
study was generally
21
study. A systematic
22
affects of regular break times, cyclical activities,
23
Raymond
24
taking video pictures,
25
observations.
26
the enumerator
27
procedure
28
magnifies the probability
time interval occurring
had his enumerators
recording
his enumeration
acceptable
frequently
of data for
for that species
enough
of
to minimize
the
However,
Mr.
was measured.
also doing a variety of other activities,
such as
paces walked, at the same time as tallying the
Tallies were given a lower priority than these other activities, with
entering the information
is unacceptable
from memory some minutes later. This
in a typical cost estimate study because
of error.
29
14
potentially
it
1
iv. Training manuals and log-books
2
Mr. Raymond
3
4
collectors
5
enumerators
8
sheets.
has consistently
said that no training manuals for his data
exist and that the only logs kept were
on the daily records that are buried in volumes
In a typical cost study all data collectors
7
the notes
course to ensure consistency
9
work with a training manual to use as a reference document
logbook
is normally
11
anomalies,
12
observations
13
well-designed
between enumerators,
kept in which work times,
are recorded
- - together
after-the-fact.
of other raw data
and each would commence
numbers
during the study. A
of observations
with any changes
These manuals
by the
would pass the same training
8
10
made
and
that are made to the
and logs are key elements
of any
statistical survey.
14
15
v. Training the trainers
16
Where it is necessary
17
for more than one trainer to be involved in training
18
the enumerators,
19
to ensure subsequent
20
load time begins at the moment that the letter carrier’s feet stop moving at the
21
end of a walk and ends at the moment that the foot is lifted to start away from a
22
stop.
23
It should
24
Raymond’s
25
26
video training,
27
the consistency
it is essential to identify the key points that must be focused on
consistent
observations
also be remembered
by the various trainees,
that the majority
study focused on factors of importance
how to enter the information
28
15
of the training
to the Engineering
for Mr.
Study, i.e.
with the bar code reader, how to
identify the various activities and types of mail receptacle
and accuracy of cost-related
e.g. the
rather than maintaining
data collection.
1
vi. Training the enumerators
2
Enumerators
3
all need to be trained to the same observational
4
data
5
Engineering
6
certainly
7
activities for industrial
8
occurrence
9
and woefully
10
are to be consistent
across
strata
standards
In Mr.
or clusters.
if
Raymond’s
Study a variety of different training methods were used, which were
acceptable
for the work being undertaken
engineering
time estimates,
- observations
frequencies,
of the work
and percentage
of various different activities. This training however, was inconsistent
inadequate
for data collectors
working
on a statistical
study to
allocate costs.
11
12
vii. Decision Rules on Data Acceptance
13
Elimination
14
of any sampled
15
cases, e.g. violent weather,
16
determined
17
clients and to other researchers
18
procedures
19
not to have been subscribed
20
data to make cost estimates.
decision
data should
only occur
in extremely
power failure and the like, and in accordance
rules.
Excluded
data are usually presented
attempting
are not necessarily
to fully by Mr. Raymond
Studies.
with pre-
for review by
to replicate the study results.
adhered to in Engineering
vexed
These
They appear
when using engineering
21
22
viii. Data Qualify Maintenance
23
Throughout
24
this discussion,
the emphasis has been on efficiently obtaining
25
usable research
results, without
26
and stakeholders
all have interests in getting the best (accurate)
27
date
28
population,
sample
statistics
sacrificing
concerning
the
in this case the route operations
16
data quality.
key cost
Researchers,
parameters
clients
and most up-toin the
parent
of the United States Postal Service.
Certainly,
1
there are many examples
where the budget or available
2
frame has driven the sample size and the confidence
3
suffered accordingly.
4
far-reaching
5
based
6
considerations
7
statistical methodologies
obtained
have perforce
Indeed,
8
9
upon results
in the results obtained
It is however, critical to recognize
cost and revenue implications
that decisions
has
that have
may not be best served if they are
from subsidiary
studies
in which
led to a series of deviations
corner-cutting
from “best-practice”
for cost studies.
the Data Quality
Study (1998)
emphasized
the importance
methodological standards rather than abandoning
improving
time-
10
leaders in postal ratemaking
11
require the highest
12
resources -to
them.
practice, the Postal Rate Commission
standards
of research
enhance its deliberations
performance
of
As world
continues
to
- given the available
and inform its decision-making.
13
14
IV.
Are any data better than no data?
15
16
i. The need for new USPS cost data.
17
There is general agreement
18
about the long-standing
19
improve the USPS cost data.
This need was highlighted
20
Data Quality Study which unearthed
21
which are still being applied
22
(USPS)
23
parameters
24
change,
25
extraneous
forces.
26
“traditional”
cost parameters
27
into” by both the USPS and the stakeholders.
(the mailers)
dating from the past two decades
productivity
shifts,
traffic
Nevertheless,
in several parts of the
“rules of thumb” dating back to the 1920’s
in the twenty-first
and the stakeholders
need to up-date and
patterns,
century.
recognize
Moreover,
the client
that the familiar
cost
have been overtaken
by technical
work
many
methods
and
other
any shifts away from these long established
should be gradual, well founded and widely “bought-
28
17
1
ii. Qualify Data for Qualify Decisions
2
In the balance,
3
there is far too much revenue/expenditure
4
ratemaking
5
procedures.
6
be done and the USPS needs to find the budget to commission
7
priority.
8
would “Big One” lottery ticket holders have thought
9
included in the recent $360 million lottery drawing?!
decisions
to be based on inadequate
A robust and scientifically
defensible
at stake for
new data or flawed research
innovative cost study needs to
it, as a matter of
Band-Aid solutions and half measures are simply not acceptable
must be available
if their numbers
- - what
were not
All the data from the parent
10
population
for a random sampling
11
vetting must be done when the research is designed,
12
upon.
process
and professional
implemented
and reported
13
14
iii. What the Data Qualify Study said about Letter Carrier Costs
15
The Data Quality Study, commissioned
16
jointly by the PRC, USPS and the
17
General Accounting
18
to Delivery Costs. Pages 53 to 56 of the Technical
19
Appendix. These recommendations
20
-
21
22
Redesign
Office, was quite specific in its recommendations
and update
with regard
Report #4 are provided in an
include:
the relatively
old and highly
imprecise
Delivery
special studies.
-
Review the data being developed
if this
information
by the Delivery
is a possible
Re-design
long-term
(my
project to
23
assess
emphasis)
24
replacement
25
These imply an extensive
26
and what the Postal Service should do with it. The recommendation
27
with the following
28
the rate making forces within the Postal Service to impact the quality of data to
29
be collected in this new system.”
for IOCS and some special study data.
important
discussion
statement:
of what the Re-design
“Reviewing
16
project was doing
was qualified
this data now can also allow
1
It most certainly
did not say: “Dig into what has already been done and
2
see if you can fit some previous
3
special
4
understand
5
reviewed by the Data Quality Study - - could be used for rate-making,
6
Kearney was responsible
7
Study managed
8
solution speaks volumes.
studies.”
In fact no-one
whether
observations
into something
to replace
the
could be better placed than A.T. Kearney
the work by Mr. Raymond
- - already
completed
to
when
since A.T.
for both the Data Quality Study and the Engineering
by Mr. Raymond.
The forward-looking
nature of the suggested
9
10
iv. Is the Engineering Study data better than no data?
11
Great caution
12
should
be exercised
in considering
Study data results as a basis for developing
whether
to use the
13
Engineering
14
ratemaking.
15
se. However, there is extreme reticence to use the reworked data from this study
16
for purposes for which it was never designed or collected.
There is no criticism here of the Engineering
No confidence
17
new cost results guiding
levels can be ascribed
Standards
to these data because
study per
no sample
18
design was made. The best we can say is that we have information
19
of preselected
20
unable to say. The resulting cost data, calculated
21
and even enjoy a degree of accuracy,
but no one can say with any confidence
22
what value to put on these
estimates
23
fashion
24
achieve,
25
replicable and scientifically defensible study of relevant cost parameters in the
26
USPS route system at the earliest opportunity.
in which
postal stations.
How these relate to the total universe
sample
how
important
19
we are
by Mr. Baron, may be indicative
because
of the unacceptable
The one thing that these
they were obtained.
is to underscore
on a number
it is to undertake
results
do
a transparent,
CERTIFICATE
OF SERVICE
I hereby certify that I have this date served the foregoing document
upon all participants of record in this proceeding in accordance
with the
Commission’s Rules of Practice.
Washington DC
May 30,200O