ESTIMATION OF EFFICIENCY OF
UKRAINIAN HOSPITALS IN THE
CONTEXT OF HEALTH REFORM
FLOW (CASE OF KIEV)
by
Roman Zaika
A thesis submitted in partial fulfillment
of the requirements for the degree of
Master of Arts in Economics
National University “Kyiv-Mohyla Academy”
Master’s Program in Economics
2008
Approved by __________________________________________________
Mr. Volodymyr Sidenko (Head of the State Examination
Committee)
Program Authorized
to Offer Degree
NaUKMA
Master’s Program in Economics,
Date
i
National University “Kyiv-Mohyla Academy”
Abstract
ESTIMATION OF EFFICIENCY OF UKRAINIAN HOSPITALS IN
THE CONTEXT OF HEALTH REFORM FLOW (CASE OF KIEV)
By Roman Zaika
Head of the State Examination Committee: Mr. Volodymyr Sidenko,
Senior Economist
Institute of Economy and Forecasting
National Academy of Sciences of Ukraine
The study analyzes the relative efficiency for city public clinical
hospitals in Kyiv for period 2001-2005. This period encompasses activities
that were directed on re-orientation of health services provision from
inpatient to outpatient care. The study was focused on measuring the
influence of transformations on improvements in hospitals' performance.
Measurements of relative efficiency were calculated by using non-parametric
Data Envelopment Analysis and bootstrap technique. The results of reorientation were expressed in terms of efficiency score and relative difference
between aggregated efficiency scores of hospital groups. The impact of reorientation efforts was found to be statistically insignificant from structural
side but areas of influence on efficiency during next step of reforms were
identified.
i
TABLE OF CONTENT
C H A P T E R 1 ..........................................................................................1
INTRODUCTION ..................................................................................1
C h a p t e r 2..................................................................................................4
LITERATURE REVIEW.........................................................................4
Data envelopment analyses development .........................................................4
Improvement of DEA methodology and development of alternatives .....................5
DEA in health care ..................................................................................7
Ukrainian studies .....................................................................................9
C H A P T ER 3 .........................................................................................11
METHODOLOGY ...............................................................................11
Theoretical framework..............................................................................13
Practical framework.................................................................................15
C H A P T E R 4 ........................................................................................20
DATA ....................................................................................................20
Health care sector description .....................................................................20
Kyiv clinical hospitals ...............................................................................25
Data limitations .....................................................................................28
C H A P T E R 5 ........................................................................................29
RESULTS OF ESTIMATION ...............................................................29
CONCLUSIONS ...................................................................................33
BIBLIOGRAFY .....................................................................................35
APPENDIX ...........................................................................................38
ii
LIST OF FIGURES
Number
Page
Figure 1: Number if medical staff in Kyiv hospitals .................................21
Figure 2: Number of middle medical staff in the hospital .........................21
Figure 3: Clinical beds ...............................................................................22
Figure 4: Number of beds (all) ..................................................................23
Figure 5: Ultrasonic tests in all hospitals in Kyiv......................................24
Figure 6: X-ray tests in all hospitals in Kyiv .............................................25
Figure 7 Dynamic of the means of variables .............................................26
iii
ACKNOWLEDGMENTS
I would like to express my appreciation to my thesis adviser,
Professor Valentin Zelenyuk for inspiring me, and guiding through world of
efficiency analysis.
I'm grateful to Professors: Tom Coupe, Olesya Verchenko Hanna
Vakhitova for providing valuable comment during my writing.
Special thanks is for Dudar Yulia and Oleg Petrenko who have
helped me with data collection.
Thanks to Orest Novoselskiy for great technical assistance
iv
GLOSSARY
DEA. Data envelopment analysis
GUOZ. Main Department of Public Health
MOH. Ministry of health
DMU. decision Making Units
SFA. Stochastic Frontire Analysis
FDA. Free Disposal Hull
v
Chapter1
INTRODUCTION
Hospitals have always been key providers of treatment
services in health care system. As a country of former USSR,
Ukrainian public health system was focused on secondary
(inpatient) medical care. After the first polyclinic attendance in
FSU countries physicians sent 25-30% patients to hospitals. (In
Great Britain the same parameter was 8.6%, in USA – 5,2%). And
65-85% of state public health finance provision was spent on
secondary treatment against 45-50% in OECD countries (Ensor
(2003).
The next step after 1991 of reforming public health sector
in Ukraine was induced by presidential decree in 2000 after which
reorganization activity were implemented in practice in 2002. One
of the main directions of actions was re-orientation of health
system efforts on outpatient (ambulatory) care rather then on
inpatient (stationary). It denoted that disease cases of lower
severity might not require hospitalization. Hospitals would be less
loaded with such cases, which in turn, would save resources of
inpatient treatment.
The efficiency of hospitals as the main chain of medical
care provision was narrowly examined in Ukraine, despite many
studies could be found in the world practice. Also, analysis that
compares the efficiencies of groups of hospitals within health care
system was not conducted before. At the same time, only a few
1
studies in Ukraine were dedicated to changes in hospitals
efficiency as those that were related to health care reform.
The objective of this research is to estimate the changes in
aggregated (grouped) efficiency score of main health care
providers in Ukraine during the period of health care system reorientation from inpatient to outpatient direction. As purpose of
this re-orientation was to reduce the consumption of resources by
inpatient treatment activity while not decreasing the treatment
services provision, this study investigates the success of achieving
this purpose in terms of efficiency score alteration and relative
differences between aggregated hospitals' efficiency scores.
Study evaluates technical efficiency score of clinical
hospitals in Kyiv for period 2001-2005 with the help of Data
Envelopment Analysis (DEA). Clinical hospitals were used
because of homogeneous production technology they had. The
obtained technical score was then aggregated in two groups. First
group - hospital score before 2002 and second group – score after
2002.
For seven model's combinations of input and output that
were analyzed, bootstrap technique was then employed to get bias
corrected aggregate score and confidence interval for each
combination. Eventually the relative difference was tested
between two aggregated groups. The relative difference for
weighted mean score of two groups was also considered.
The rest of the paper is organized as follows: the Literature
review section gives the overview of previous studies on
methodology and its application in hospitals. Methodological
2
section depicts the theoretical and practical frameworks used.
Data section describes the information related to research. Section
Results illustrates the results of conducted analysis and discuses
them. The final section concludes.
3
Chapter2
LITERATURE REVIEW
The description of previous study was divided I two main
ways. First, I have focused on description of basic methodology
development that was Data Envelopment Analysis (DEA).
Secondly, I have stressed my attention of a studies dedicated to
methodology application in public health field. Thirdly, I have
paid attention to the relevant Ukrainian literature.
Data envelopment analyses development
One of the techniques of estimation of the behavior of
decision-making units (DMU) is to measure their productive
efficiency. Until now economist have developed many methods to
perform this task, however the first comprehensive measure of
economic efficiency applicable both on micro and macro levels
dates back to Farrell (1957) where he not only proposed “to
compare the performance with best actually achieved rather than
with some unattainable ideal” (theoretical yardstick) but also built
evaluation model for multiple inputs. Earlier Malmquist (1953)
has suggested to use indices to measure change in productivity
over time thus adding dynamic to results of evaluation. Farrell's
work has been taken as a research tools in many production
industries and public sector giving a push for development of
other productivity evaluation methods.
4
Particularly, estimation of the efficiency in public health has
necessitated specific methods developed for such purposes. One
of them was data envelopment analysis (DEA), the nonparametric
method based on the use of mathematical tool of linear
programming developed by Charnes, Cooper and Rhodes (1978).
It allowed making a comparative estimation of efficiency in health
care system taking into account different sets of resources and
production. Since that DEA has become the most widely used
methodology to measure the performance of decision-making unit
inside economic systems.
Further studies devoted to research in estimation of
efficiency in health care may be split in two ways. First - dedicated
to improving the base methodology (DEA) and second that deals
with applications of DEA in public health industry under different
circumstances, data sets, and results.
Improvement of DEA methodology and development of
alternatives
Following the first way will lead us to work of Cherchye et
al. (2000a) who found that convexity assumption original
methodology could be dropped to make computation easier.
The
same
authors
(2000b)
introduced
so
called
“nonparametric test statistics (efficiency depth) for testing for
efficiency in case of errors in variables” which became one of the
method to smooth the problem of DEA limitation – big
sensitivity to data errors.
Wilson (1993, 1995) tried to find the “diagnostic tool” for
this DEA weakness when data were with errors. He has stated
5
that measurements error and outliers should be corrected if
possible or otherwise observation may be dropped by researcher.
In the same year Hall, et al.(1995) suggested iterated
bootstrap method for error correction in frontier models. Later
Hall and Simar (2002) approached to problem more practically
using Monte Carlo simulated data relaxing the assumption that
error distribution is known. In 2000 a general methodology for
bootstrapping in DEA model was created by cooperative effort of
Wilson and Simar (2000).
As alternative to DEA, other non-parametric envelopment
estimators like Free Disposable Hull (FDH), order-m was used.
For a specific observation and output oriented model all other
observation that are more efficient in input are selected. Then
from such a group several samples of size m are drawn with
replacement. At the end the expected output maximum by m
firms using input x is used as benchmark. They were briefly
summarized by Simar (2002) who showed that FDH was easier to
compute, did not need convexity assumption and at the same time
could be consistent. He also pointed out that if, “additional
statistical noise and imprecision are created while estimating the
unknown quantities with DEA/FDH the bootstrapping is an
alternative to them”.[24]
Fare and Grosskopf (1985) stressed the attention that the
return to scale which technology exhibits was important in
measuring the efficiency. They suggested to compare the DEA
estimators under Constant return to scale (CRS) various return to
6
scale (VRS) and non-increasing return to scale (NIRS). Thus
results of a score could be different depending whether the
estimated model is input or output oriented.
Many of the studies compared DEA to parametric method
called Stochastic Frontier Analysis (SFA) (Jacobs (2001),
Hollingsworth (1998)) which allows for statistical “noise”. Unlike
Jacobs, Hollingworth does not tells exactly which type of method
to use for estimating hospitals efficiency still he concentrates
specifically on DEA application in health care.
We now see that many theoretical studies were devoted to
elimination of disadvantages of DEA, to development of
consistent non-parametric substitutes and compliments of DEA
score. Very fast DEA was integrated into analysis of operations of
the firms, treatment establishments, banks. In practice, it helped
to define where to allocate the scare budget recourses of a
company.
DEA in health care
Second part of literature review is about studies in public
health sector using DEA begins with Hollingsworth who has
published the application of DEA in health care in 1983. Many
works have been done at that field thereafter. But as later the
same author pointed out most of the studies were dedicated to
measuring technical efficiency of hospitals and little attention was
paid to allocative efficiency (e.g. Sengupta (1998)). It was Farrell
(1957) who has first introduced the decomposition of overall
efficiency consists of technical and allocative efficiencies. After all
Worthington (2004) came back to question of choosing the most
7
appropriate methodology together with model specification. He
has analyzed studies from 1983-2003 done throughout the world
and found that neither DEA nor SFA as well as different
specification of input and output units could gave a complete
picture of overall efficiency. But he made noteworthy conclusions
that another health financing system, when used, “improves
efficiency over the budget-based allocation of funds, and as a
result, reforms in health system funding have mostly improved
allocative, rather than technical, efficiency. Finally, it is also the
case that the efficiency of health care organizations and industries
has improved over time.” Sengupta (1998)
Besides selection of DEA modification, the selection of
inputs and outputs is important because distribution of final score
will depend on that. Specifications (mostly in physical units or in
costs expressed) of input and output models were surveyed in
above mentioned work by Worthington (2004) but distinctively
Mersa (1989) has marked out two approaches for output selection
by treatment establishments:
process approach based on the notion that hospitals output
is nothing else than procedures made by different departments: Xrays, different tests, patient days and others
- outcomes approach based on the notion that procedures
are only middle link of chain to desired patients’ health
status
Recently, Ferriel at al. (2006) investigated so-called
hospitals “output congestion” - (uncompensated expenses) and
found that uncompensated care reduced the production of other
8
hospital outputs by 2%. “Thus, even if hospitals were to operate
efficiently, they might still face financial distress as a result of
providing uncompensated care.” Ferrier et. al. (2006)
Dexter et al. (2002) came from the other side and tried to
specify more properly the input areas. They used physical unit
(number of beds, physicians) and technology index the value of
which depends on number of the particular services provided by
the hospitals (cardiac surgery, urological surgery etc). Such index
could be a proxy to weight heterogeneous hospitals.
Fare, et al., (1995) on that example of Swedish
Pharmaceuticals industry emphasized that quality changes is
important factor of productivity change. Then Chun (2006)
described that there were three ways in which quality of medical
service provision is influence productivity namely, “the structural
or medical care input (Number of physicians, beds), the process
of providing medical services and the results or effects of medical
treatment”. He found that structural change consideration was
appropriate to Taiwanese health care system that also has been
reformed.
Conclusively,
adaptation
of
efficiency
estimation
methodology to specific purposes of health care had and has been
proceeding.
Ukrainian studies
Despite widely used in the world practice, little attention
was drawn to estimation of efficiency to Ukrainian hospitals.
Pilyavsky and Valdmanis (2002) were first who have
estimated the efficiency of for hospitals in Ukraine with non-
9
parametric approach. They have examined differences in
healthcare efficiency between western and eastern oblasts and
have explained differences in behavior as those that could be
related to cultural biases.
Then Pilyavsky and Staat (2006) have studied the efficiency
of Ukrainian hospital and policlinics in Ukrainian. They
investigated how the productivity has been varying over time
from 1997-2001 and found changes in productivity only in period
from 2000 to 2001. It was supposed that these changes might be
due to Reform Plan introduction in 2000 but there was no
evaluation of changes of efficiency thereafter.
In 2008 Pilyavsky et. al (2008) came back to estimation of
differences between eastern and western oblast. The study was
concentrated on polyclinics during period 1997-2001 and found
eastern units more efficient. They explained those with differences
in health budget and demographic characteristics.
As a summary of the reviewed literature, following
facts may be outlined:
-
DEA is well-developed approach to study relative efficiency
but it is not relieved from limitations which could be
mitigated.
-
The non-parametric methods in productivity analysis were
not widely applied to Ukrainian domestic hospitals while in the
world it is most frequently used technique.
10
C h a p t er 3
METHODOLOGY
In this chapter I described the method I have chosen for
research and the reason for my choice. The empirical framework
was also provided.
Traditionally there were two main directions to evaluate the
efficiency of observed units. One of them was approach based on
average value in the sample e.g. expected rate of return of
financial portfolios. Another direction dealt with relative
efficiency within the sample where observations were compared
relative to each other but not to specific value (frontier analysis).
When we talk about countries in transition, usually the
price set for medical services in public hospital is unknown.
Consequently, when price is unknown, the method of efficiency
estimation that operates only with natural values of output and
input is more appropriate.
Over the last twenty years Data Envelopment Analysis
(DEA) has been more and more actively applied (Jacobs, 2001).
DEA is the nonparametric method based on the use of
mathematical tool of linear programming. It allowed to make a
comparative estimation of efficiency in the health care system
taking into account different sets of resources and production
made.
I used DEA for my study because comparing to regression
method it has the following advantages:
- it is non-parametric and does not require specific functional
form to be placed on hospital production technology.
11
- it allows the use of multiple input and multiple output and
makes the aggregation of efficiency score is possible. In addition it
eliminates multicolinearity problem
- it is a frontier approach. Unlike the statistical analysis, it does not
require various hypothesis of parameters testing.
- it is less sensitive to small number of observation
Taking into account the methodology advantages and
literature overview finding the hospital efficiency score was
calculated at the first step of analysis. At second step the
efficiency score was aggregated in 2 groups. One group for years
2001 and 2002, send for 2003-2005.
Aggregate efficiency score could be compared using
bootstrap-based test of equality of aggregate efficiencies. For
group 1 and 2 the relative difference (RD1,2)
in
aggregate
technical efficiency of ( TE )was tested.
___
H0: TE1  TE 2
Then
for
DEA
vs.
H1: TE1  TE 2
estimator: RDˆ 1, 2  T E 1 / T E 2 .
If
the
confidence interval for given ratio appears to be out of unity the
difference would be statistically significant. The logic presented
was developed and described in Simar and Zelenyuk (2005).
Zelenyuk and Zheka (2004).
However before comparison it is necessary to derive the
DEA bias corrected efficiency score with the help of bootstrap
approach as was suggested by Simar and Wilson (2000a, 2006).
Then a bootstrap analogue of relative difference ration
12
is RDˆ *1, 2,b  T E *b / T E *b , b=1,…B. Where b is a number of bootstrap
1
2
iterations.
Further, for the purpose of identification of the factors that
may influence the change in efficiency the truncated regression
method may be employed. (Simar and Wilson 2006). Corrected
efficiency score is regressed on several exogenous factors (e.g.
health budget, demographic situation). The significance of factors
would suggest them as souses of efficiency change.
Now we came to the description the theoretical framework.
Theoretical framework
For
K
hospitals
(k=1,…K)
that
produce
output
y  ( y1 ,..., y M )  RM , using input
x k  ( x1 ,..., x N )  RN we have technology set

T  ( x , y ) :" y _ is _ producable _ from _ x "

and
equivalently

P ( x)  y : ( x , y )  T
Then
,
x  RN
assuming that technology satisfies standard regularity
axioms:
Axiom 1. “No free lunch”: y  P k (0 N ), y  0 M
Axiom 2. “Producing nothing is possible”: 0 M  P k ( x), x  RN
Axiom
3.
“Boundness
of
the
output
set”:
P k ( x) _ is _ a _ bounded _ set , x  RN
Axiom 4. “Closeness of the Technological set T”: Technology set Tk
is a closed set
13
Axiom
5.
“Free
disposability
of
outputs”:
y 0  P k ( x)  y  P k ( x), y  y 0 , x  RN
we have DEA estimator of Farrell output oriented technical
efficiency (TE) score of observation j (j=1,..n)
TE ( x j , y j )  max 
 , z1 ,... z 2
s.t.
n
z
k
y mk  y mj ,
m=1,…M
k
x mk  xymj
i=1,..N
k 1
n
z
k 1
  0, _ z k  0
k=1,…n
where  is measure of distance to the frontier and z k is an
arbitrary scalar via which the radial expansion or contraction is
made within technology set T. z k  0 also presents the variable
return to scale. (If z k =1 we have constant return to scale (CRS)).
The Variable Return to Scale (VRS) is used because we estimate
model for short-run however for research purposes we may also
impose CRS into model to compare it to VRS specification. The
output orientation was chosen as we are interesting in
maximization of medical service provision under given level of
resources.
In order to get aggregation score the technique of price
independent weights is applied. Their bootstrap analogues are
(Simar, Zelenuyk (2005).
______
*l
b
s
TEˆ  kl1 TEˆ b*l ,k  S b*l ,k ,
where S b*l ,k 
14
1
M
y ml ,k
M

m 1

sl
k 1
y m*l,,bk  S bl
and
______
*
b
L
TEˆ  l 1TEˆ b*l  S b*l ,
1
where S 
M
*l
b
y ml ,k
M

m 1
 
L
sl
l 1
k 1
y m*l,,bk
where
b=1 ,…, B is number of bootstrap iterations
k=1 ,…, n – number of firms in original sample
k=1 ,…, s – number of firms in bootstrap sample s<n
l=1, …, L – number of groups; m=1,…M – outputs of the
hospital
For the bootstrap itself the algorithm is following:
For each hospital in the sample   ( x k , y k ) : k  1,...n we
obtain TEˆ ( x k , y k ) : k  1,...n calculated score.
Then aggregate the score and obtain the bootstrap sample

 *  ( x *k , y *k ) : k  1,...sl
 to
get bootstrap estimate TEˆ b*l ,k . After B
 
iterations we get estimates for aggregate efficiencies TEˆ b*l
 
group l and for entire group TEˆ b
B
b 1
B
b 1
of
and use these estimates to
construct the confidence interval for biased corrected DEA
estimates (Simar, Zelenuyk (2005).
Practical framework
The measurement of hospital output could be reflected by
a volume of services that, if consumed, might improve health
15
status. The specific to hospitals operational measures like length
of treatment also characterize output.
The hospitals input
measures could be expressed in terms of capital (beds), labor
(medical staff) as well as in financial terms (wages, fee for
services). See Afonso (2008), Chun Lui (2006).
In order to control for specification sensitivity and taking
into account Worthington (2004), Pilyavsky and Staat (2006),
seven models with different input and output combinations were
constructed. They are presented at table 3.1.
Table 3.1 : Model specifications used for analysis
Model
1
2
Input
Output
Description
Medical staff (physicians and nurses)
Number of patients discharged
All structural
Beds in use
x-ray test
variables
Days of temporary disability
ultrasonic test
Medical staff (physicians and nurses)
Number of patients discharged
Labor and capital
Beds in use
x-ray test
input only
ultrasonic test
3
4
Days of temporary disability of
Number of patients discharged
Proxy for financial
physicians
x-ray test
input. Bed utilization
Working period of beds
ultrasonic test
Physicians
Number of acute surgeries
Only treatment
Working period of beds
Number of patients discharged
services as output
Staff per beds ratio
Length of Treatment
Quality components
5
x-ray test
ultrasonic test
Staff per beds ratio
6
Length of Treatment
Quality as input.,
Number of acute surgeries
Operational
components
7
Days of temporary disability of
Length of Treatment
16
Proxy for financial
physicians
input.
Working period of beds
Operational
component
It worth mention that because of lack of financial data the
model specification was limited to structural characteristics of
hospitals. However proxy (Days of temporary disability of
Physicians) was developed and used to address the financial
information deficit. This one and the rest of the variables are
describes in table 3.2.
Table 3.2: Description of variables
Name
Units
Description
Inputs
MEDICAL
STAFF
Number
Characterizes the labor input. This is
of persons
main
factor
of
treatment
services
provision in inpatient care.
Number
Capital side of a hospital. Reflects the
capacity. All beds under study had clinical
BEDS IN USE
(acute) care profile.
Day
Reflects the number of days per year that
physicians of particular hospital were sick
DAYS OF
TEMPORARY
DISABILITY
or missed the work day by other reason.
The big number of days indicates that
hospital bears expenditures but services
were not being delivered at those periods.
WORKING
PERIOD OF
BEDS
STAFF PER
BEDS RATIO
Days
Represents the utilization of capital per
year. The higher number of day per year
the higher utilization is.
Ratio
Modification of structural component.
17
May influence efficiency in both ways, to
increase it as bigger number of doctors
may deliver more services and to decrease
it because many physicians imply higher
budget spending.
Ratio also presents
structural quality component. Chun Lui
(2006)
Outputs
NUMBER OF
PATIENTS
DISCHARGED
Number
The
direct
output
of
hospitals.
of persons
Discharged patient are main consumers
of inpatient treatment services.
Number
This component of hospital output
captures diagnosis activity. While it is not
a treatment itself it establishes and
X-RAY TEST
confirms the indication to the type of
care.
ULTRASONIC
Number
TEST
This component of hospital output
captures diagnosis activity. While it is not
treatment
itself
it
establishes
and
confirms the indication to the type of
care
NUMBER OF
Number
It is purely treatment activity of a
ACUTE
hospital. Characterizes the number of
SURGERIES
services that hospitals deliver with given
capacity (labor and beds). All surgeries
under study were of clinical (acute) type
to provide homogeneity of production.
LENGTH OF
Days
TREATMENT
staying in lower value may lead to increase in
bed
Illustrates the quality of treatment. The
turnover of patients during the year thus
positively affecting productivity of a
18
hospital.
The inverse values of this
variable
was
taken
for
output
maximization.
The next section will describe the data that was used for
test the model. But before going further the methodology
disadvantages have to be indicated in order to logically end this
section up.
Methodology disadvantages
- The inherent feature of DEA was the assumption of
homogeneous of technology of hospitals. To assure the
homogeneity the clinical hospitals from Kiev Main Department of
Public Health (GUOZ) system were taken as a sample.
- The problem of endogeneity bias still exists.
- When using non-parametric techniques statistical
hypothesis tests are difficult to conduct.
- The DEA score may be downward and is inappropriate
to be used during second stage analysis. The corrected efficiency
score could be achieved through bootstrap process. Simar, Wilson
(2007)
- In our case technical efficiency scores refer to relative
performance within the sample. Hospitals given an efficiency
score of one are efficient relative to all other hospitals in the
sample, but may not be efficient by some absolute or world
standard necessarily.
19
Chapter4
DATA
In this section the data used to for analyses was described.
The chapter was divided in two main parts. First one gives brief
overview of Kyiv health care sector during the period under
study. Second one describes performance of Kyiv clinical
hospitals.
Health care sector description
Kyiv as a capital of Ukraine has one of the best developed
public hospitals infrastructure (specialists, availability). Still pattern
of changes in efficiency may not be similar to other areas of the
country so results should be extrapolated at the entire Ukraine
with big caution. To be familiar with situation that took place in
Ukrainian health sector as well as in Kyiv the dynamic of hospitals
input and output was illustrated on the following diagrams.
20
Figure 1: Number if medical staff in Kyiv hospitals
Number of medical staff (Physicians). Kyiv hospitals
14000
12529
12916
12863
12692
12591
12000
10000
8000
5254
4624 4103
6000 4172
5575
5382
4298
4338
4000
4411
2000
0
2001
2002
2003
MOH system
2004
2005
GUOZ system
Other
Figure 2: Number of middle medical staff in the hospital
Number of Middle medical stuff. Kyiv hospitals
30000
25000
23244
22815
22521
22660
21310
20000
15000
10000
5000
4029
3766
7862
7564
7200
4172
7796
4349
3908
0
2001
MOH system
2002
2003
GUOZ system
2004
2005
Other
Hospitals under study belong to Main Department of
Public Health system. Two labor indicators physicians and middle
stuff show slight but opposite movement. The slow movement up
of a number of physicians may be explained by the high demand
for working place among medical school graduates and specialist
from close to Kyiv area due to financial (payment from patients)
21
and professional opportunities. Although the numbers of work
places are limited by number of wages that local budget finances.
At the same time Middle staff (Nurses) positions has been
reduced. The explanation goes to low financial opportunities of
the Nurses' job.
The next diagram present the capital side of the hospitals –
beds. For this research the beds of obstetrics, rehabilitation and
other non -acute specialization were excluded. Instead, beds with
surgery, including children care, therapy profile were considered.
Figure 3: Clinical beds
Beds of clinical profile in Kyiv (surgery, therapy, children care)
12000
11840
Number of beds
11500
11000
10830
10500
10452
10342
10317
2002
2003
10538
10000
9500
2000
2001
2004
2005
bed of clinical profile
It is noticeable that number of beds of such type was
decreasing till 2002. But if we consider the next picture of a
number of beds in all treatment establishments of Ukrainian
capital, we may detect no considerable change in the entire system
with one exception after 2001. The re-orientation from inpatient
care to outpatient stated by Reform plan in 2000 elucidates the
22
change of proportion of inpatients beds. At the same time as
number of hospitals together with beds can not be reduced
according to Constitution of Ukraine the total number of beds
was saved.
Figure 4: Number of beds (all)
Number of beds in different tretment establishment in Kyiv
35000
30000
30330
29681
29362
29768
29563
25000
20000
19202
18752
18762
18828
18820
15000
10000
0
1490
2001
Total
5127
4740
2841
5007
3031
5000
1490
2002
5127
2757
2579
1532
1490
2003
GUOZ and MZ
5049
2729
2004
MOH
1532
2005
NASU
Other
The important component of hospitals activity is diagnostic
procedures. They are described at the next two diagrams
23
Figure 5: Ultrasonic tests in all hospitals in Kyiv
Ultrasonic tests in all hospitals per 1 000 pupulation
600
579,1
518,7
500
527,2
481,8
481,0
430,8 432,4
389,1
400
397,2
386,1
349,9
356,8
317,6
300
289
271,1
307
248,7
208,5
200
329,4
264,8
241,4
221,7
286,2
267
235,8
201,9
215,3
192,7
185,2
100
1996
1997
1998
Kyiv (territory)
Ukraine
1999
2000
2001
2002
2003
Kyiv (GUOZ MZ)
24
2004
2005
Figure 6: X-ray tests in all hospitals in Kyiv
X-ray tests in all hospitals per 10 00 population
6000
5777,7
5500
5385,2
5025,4
5000
4635,5
4500
5103,8
5049,7
4593,2
4632,0
4273,2
4406,6
4444,6
4091,0
4029,7
4000
3510,8
3550,7
3500
3397,7
3046,8
3000
4011,4
3892,7
3617,2
3584,9
3742,9
3859,8
3428,3
3230,9 3278,4
3035,8
3011,5
2860,2
2500
1996
1997
1998
1999
2000
Kyiv (territory)
Ukraine
2001
2002
2003
2004
2005
Kyiv (GUOZ MZ)
Despite the "Number of diagnostic tests" in Kyiv hospitals
exhibits continuous growth it is possible to notice the growth
slow down after 2000 and 2002 – years of reform unfold. Still
those changes may have two-side effect on efficiency. One is
negative, as hospital output decreases. Another is positive as
hospitals spend fewer resources on tests performance. We will
investigate further the changes in hospital efficiency when these
variables were used.
Now let us go further and investigate the performance of
hospitals under study.
Kyiv clinical hospitals
Eighteen Kiev city clinical hospitals from Main Department
of Public Health (GUOZ) system were taken. The number of
25
observations used to evaluate the efficiency during 5 years has
totaled to 90. The data set for each year was taken from annual
GUOZ report and was provided by "ATRIA" organization.
The descriptive statistic for variable used for model
specifications is presented in tableA1 in Annex1. It compliments
the description of components started in methodology chapter.
While in the graphs given below the dynamic of mean values of
variables is presented.
Figure 7 Dynamic of the means of variables
Days of temporal disability (mean)
1400,00
Working period of bed per year
(mean)
1350,00
1300,00
325,00
1250,00
320,00
1200,00
315,00
1150,00
310,00
1100,00
305,00
1050,00
1000,00
2001
2002
2003
2004
300,00
2001
2005
2003
2004
2005
Working period of bed per year
Days of temporal disability
Medical staff (mean)
520,00
515,00
510,00
505,00
500,00
495,00
490,00
485,00
480,00
2001
2002
Number of acute surgeries (mean)
880,00
860,00
840,00
820,00
800,00
780,00
2002
2003
2004
760,00
2001
2005
Medical staff
2002
2003
2004
Number of acute surgery
26
2005
Patients discharged (mean)
Length of treatment (mean)
19000,00
10,90
18500,00
10,80
18000,00
10,70
17500,00
17000,00
10,60
16500,00
10,40
16000,00
10,30
15500,00
2001
10,50
2002
2003
2004
10,20
2001
2005
Patients discharged
2002
2003
2004
Length of treatment
The value of days of temporary disability was growing till
2003 but diminished by 10,6% during 2004 before going up
further. The same behavior was demonstrated by the Working
period of beds with 1,4% declining at the same time experiencing
3% growth between 2002 and 2003.
Contrary to that Days of temporary disability behavior the
Number of Medical staff went up to 513,5 persons on average in
2004, which was 3,06% comparing to 2003. The number of acute
surgeries quickly began to fall after to 2002 but increased after
2004 whilst the number of Medical Staff was going down.
The Number of patients discharged has been persistently
growing while Length of treatment plummeted by 3,77% from
2003 till 2005. The lower treatment period leads to fewer days
spent in bed that in turn increase number of patients discharged
throughout the year due to faster turnover.
Five out of six illustrated variables have demonstrated
reaction it terms of value change during 2002-2004. So the
efficiency of second group of hospitals (from 2003-2005) will
27
2005
obviously be changed but we are not concerned at this moment
whether the changes were significant. We have also to keep in
mind that we refer to relative efficiency and take into account the
external factors (e.g. changes in health behavior, health insurance
market growth) that might influence the hospital efficiencies in
both positive and negative ways.
Data limitations
One more point should be noted. Besides looking at the
growth in percentage expression we should track the absolute
values. For instance length of growth altered only by 0,3 days
from 2003 to 2005 on average. Thus contribution of a variable
may happened to be quantitatively week. Thus I used this variable
as an output quality measure. Chun Lui (2006).
The number of observation was not so big, limited by
available data. Still there was no contradiction with other empirical
studies. See Afonso (2008), Bernet Pilyavsky (2008).
The lack of financial data like physicians wages, fee for
surgery, beds depreciation, official costs of treatment service,
hospital budget limits covering of entire picture of hospital
activity.
28
Chapter5
RESULTS OF ESTIMATION
By employing "Matlab 2006" software and program for
calculation of aggregated efficiency score the estimation results
were obtained for each our combination of inputs and outputs.
The number of bootstrap iteration was 500. The output
orientation was imposed. However results for input orientation
were also calculated. As public hospitals are non-for-profit
institutions it difficult to treat them as profit maximizers of costminimizers. Bernet et.al (2008). Variable return to scale (VRS) was
assumed. According to economic theory it is considered that
firms operate at optimal return to scale which is possible in a long
run. But our study is concentrated on efficiency in a short-run.
Results of calculations are presented in Annex B and
Annex C while the illustration is given to two summary tables for
output oriented model 2 and 4 interpretation.
Table 5.1 Empirical results. Model specification 2
Model2
Bias corrected
estimator
90% Confidence
interval
95% Confidence
interval
99% Confidence
interval
0,043
0,029
0,026
Lower
bound
1,043
1,019
1,035
Upper
bound
1,185
1,118
1,124
Lower
bound
1,035
1,018
1,033
Upper
bound
1,194
1,123
1,131
Lower
bound
1,023
1,013
1,027
Upper
bound
1,227
1,129
1,143
1,136
1,089
1,109
0,050
0,037
0,034
1,056
1,022
1,043
1,210
1,145
1,157
1,046
1,019
1,039
1,219
1,150
1,162
1,034
1,013
1,031
1,280
1,162
1,176
1,038
0,045
0,967
1,114
0,953
1,120
0,939
1,152
Score
StDiv
AgEffb1
AgEffb2
EntAgEf
1,113
1,072
1,087
meaneffb1
meaneffb2
OverMean
RD_AgEf
29
RD_mean
1,044
0,050
0,968
1,122
0,956
1,139
0,943
1,156
Table 5.2 Empirical results. Model specification 4
Model4
Bias corrected
estimator
90% Confidence
interval
95% Confidence
interval
99% Confidence
interval
0,042
0,049
0,031
Lower
bound
1,021
1,057
1,057
Upper
bound
1,148
1,211
1,164
Lower
bound
1,011
1,054
1,053
Upper
bound
1,162
1,239
1,169
Lower
bound
1,005
1,049
1,036
Upper
bound
1,220
1,285
1,188
1,088
1,182
1,143
0,044
0,077
0,049
1,022
1,076
1,070
1,165
1,313
1,231
1,018
1,066
1,057
1,177
1,343
1,257
1,006
1,059
1,048
1,210
1,384
1,270
0,964
0,925
0,056
0,068
0,876
0,813
1,051
1,023
0,851
0,797
1,061
1,056
0,831
0,780
1,077
1,070
Score
StDiv
AgEffb1
AgEffb2
EntAgEf
1,081
1,123
1,103
meaneffb1
meaneffb2
OverMean
RD_AgEf
RD_mean
"AgEfb 1" and "AgEfb 2" means aggregated (weighted)
score of first and second groups of hospitals efficiency
(bootstrapped estimator) respectively. "EntAgEff" is entire
efficiency of a group, how far they are from ideal. "Meaneffb1"
and
"Meaneffb2"
are
non
-weighted
mean
efficiencies.
"RD_AgEf" are "RD_mean" are ratios of two aggregated group
scores and two mean scores respectively (see Ch.3). What we are
looking at are the distances of relative difference (RD) scores
from unity and distinction between weighted and not weighted
Relative difference. Both models have slight difference between
"RD_AgEf" and "RD_mean". It is explained by the fact that
process for generating data was similar for both groups; public
hospitals do not change their type. It indicates that inpatient
hospital have stayed inpatient, with re-orientation on outpatient
30
care induced by health reform having had no significant influence.
But we should keep in mind the available data restriction that
limits characterization of hospitals from financial side.
We may also notice that RD score in Model 2 are below the
unity (group 2 was more efficient) while ratio in the Model 4 is
above 1 (group 1 has performed better). Obviously model
specification influences the results. Seven model specifications
have shown the following results of RD statistic.
Table 5.3: Relative difference score
Output
1,023
Input
0,975
RD_mean
1,025
0,982
Model 2
RD_AggEff
1,038
0,974
Labor and capital input only
RD_mean
Model 3
RD_AggEff
1,044
1,024
0,976
1,009
1,013
1,013
0,964
1,013
0,925
1,015
0,994
1,021
0,988
1,022
0,972
1,051
0,973
1,043
0,957
1,033
Model 1
All structural variables
Disability days. Bed
utilization
Model 4
Only treatment services as
output
Model 5
Staff per beds. Diagnosis in
output
Model 6
Staff per bed, Surgeries
Model 7
RD_AggEff
RD_mean
RD_AggEff
RD_mean
RD_AggEff
RD_mean
RD_AggEff
RD_mean
RD_AggEff
31
Disability days. Working
period of beds. Length of
treatment in output
RD_mean
0,954
1,037
Combinations 1,2,3 showed that efficiency of second group
was lower and Models from 4-7 showed the opposite. First 3
models took into consideration the size and utilization of capital.
Model 4 included only components of treatment activity. Models
from 5 to 7 were concentrated on operational components. So
the changes that had positive impact on relative efficiency after a
year when reform unfolds were observable and touched mostly
hospital operational activity. But hospital before 2002 used
present capital efficiently (in relative terms). However we may not
be quite sure about the source of those changes or about impact
that "consumed" those changes.
Despite the RD statistics for all 7 models lies inside
confidence intervals (Annex B and C)
and we did not find
statistically significant change we may still explore the variation in
efficiency. Second stage regression analysis, not exercised in this
work because of lack of data would be able to explain causes of
changes better taking into account influence of environmental
factors on relative efficiency change. This analysis would
contribute to the future research.
32
CONCLUSIONS
The study analyzed the relative efficiency changes for 18
city public clinical hospitals in Kyiv for period 2001-2005. This
period encompasses activities that were directed on re-orientation
of health services provision from outpatient to inpatient.
The study was focused on influence of transformations on
hospitals' performance. Relative efficiency was calculated by using
non-parametric
method
DEA
and
bootstrap
correction
techniques. The results of re-orientstion were expressed in terms
of efficiency score and relative difference between aggregated
score of hospital groups. The impact of re-orientation efforts on
hospitals efficiency was not significantly noticeable from
structural side.
There are several reasons to explain the situation. The main
one lies in political instability inside the country. Frequent
replacement of Ministers of Health reduced the motivation to
control and be responsible for process flow. Due to uncertainty
there were also no incentives for hospitals to perform in long
term action planning manner.
One
of
the
most
notable
causes
of
structural
transformation inertia was the 49 article of Constitution of
Ukraine that say that the number of treatment establishments is
fixed and can not be shorten. This means the number of beds can
not be shorten either.
33
Study concerns only one big city. So results are not to be
extrapolated straightforwardly on the entire Ukraine.
The research may contributed to policy makers through
indication of areas of potential influence during next step of
reforms; to international donors that study the efficiency of AIDS
centers and hospitals of
Sexually Transmitted Infection
treatment; and to NGOs as a part of estimation of efficiency of
field projects that were conducted by them.
34
BIBLIOGRAFY
Afonso, A., Fernandes, S., (2008) Assessing
Cherchye, L., Kuosmanen T.and Post T,
Hospital Efficiency: Non-parametric, Evidence for
(2000), "Why Convexify? An Assessment of
Portugal
Convexity Axioms in DEA", Helsinki School
Working
Papers
2008/07,
http://ssrn.Com/abstract= 1092135
of Economics and Business Administration
Working Paper W-270
Bernet, P., Rosko, M., Valdmanis, V., Pilyavsky,
Cherchye, L., Kuosmanen T., Thierry Post,
A., Aaronson, W., (2008) "Productivity efficiencies
(2000). "New Tools for Dealing with Errors-in-
in Ukrainian polyclinics: lessons for health system
Variables in DEA," Public Economics Working
transitions from differential responses to market
Paper Series” Katholieke Universiteit Leuven,
changes", Journal of Productivity Analysis., vol.
Centrum voor Economische Studiën, Working
29(2), pp. 103-111.
Group Public Economics.
Chun, L., (2006) “Measuring the Productivity of
Biorn, E., Hagen, T. P., Iversen T., Magnussen
Physicians and Evaluating the Factors that Influence
J., (2002) “The effect of activity-based financing
such Productivity in Taiwan.” Economics Bulletin,
on hospital efficiency: A panel data analysis of
Vol. 13, No. 3 pp. 1-15
DEA efficiency scores 1992-2001” Health
Economics
Research
programme
at
the
University of Oslo Working Paper 2002: 8
Dexter F, Lubarsky D.A., Blake J.T. (2002),
“Sampling error can significantly affect measured
hospital financial performance of surgeons and resulting
operating room
Charnes, A., Cooper W.W. and Rhodes E.
time allocations”,
Anesthesia-
analgesia vol. 95, pp184–188.
(1978), “Measuring efficiency of Decision Making
Units”, European Journal of Operational
Ensor, T.,(1997) “Options for health care sector
Research, 2, pp.429-444.
funding”, An introduction to Health Economics
for Eastern Europe and the Former Soviet
Union, Chichester: John Wiley.
35
Ensor, R., and Langenbrunner, J. (2003),
Hollingsworth, B.,
“Allocating resources and paying providers, in
Maniadakis N, (1998) “Efficiency measurement
M. McKee, J. Healy and J. Falkingham (eds)”
of health care: a review of non-parametric
Health Care in Central Asia, Buckingham:
methods
Open University Press.
Management Science Volume 2, Number 3
Fare, R., Grosskopf, S., 1985. “A non-parametric
Jacobs R. (2001) “Alternative Methods to Examine
cost approach to scale efficiency”, Scandinavian
Hospital Efficiency: Data Envelopment Analysis and
Journal of Economics 87, 594-604
Stochastic
and
Dawson P.J, and .
applications”,
Frontier
Analysis”,
Health
Health
Care
Care
Management Science, vol. 4, N2, pp.103–115,
Fare, R., S., S. Grosskopf, P. Roos and S. C.
34
Ray, 1995. Productivity and Quality in Swedish
Pharmacies; Comments on Productivity and quality
Lekhan V, Rudiy V, Nolte E. (2004) “Health care
changes in Swedish pharmacies; Reply, International
systems in transition: Ukraine.” Copenhagen,
Journal of Production Economics, 39(1&2):
WHO Regional Office for Europe on behalf of
137-147.
the European Observatory on Health Systems
and
Farell M.J. (1957) “The Measurement of productive
Policies,
pp97-102.
Retrieved
from:
www.observatory.dk
Efficiency”, Journal of the royal Society, Series
A(General), vol. 120, Issue 3, pp. 253-290
Malmquist S., (1953) “Index numbers and
indifference surfaces” Trabajos de estad b istica,
Ferrier G., Rosko M., Valdmanis V. (2006)
vol. 4, pp. 209-242
“Analysis of uncompensated hospital care using a
DEA model of output congestion”, Health Care
Mersha, T. (1989) Output performance measurement
Manage Science (2006), vol.9, pp. 181–188
in outpatient care. OMEGA International Journal
Hall, P., Hardle W., Simar, L., (1995) “Iterated
of Management Science. 1989 vol. 17:159–167
bootstrap with application to frontier model”
Journal of Productivity Analysis, 6, 63-76
Pilyavsky, A., Valdmanis, A. (2002) “East versus
West. Does it Make a Difference to Hospital
Efficiencies in Ukraine?”
Hall, P., Simar, L., (2002) “Estimating a
Changepoint, Boundary, or Frontier in the Presence of
Pilyavsky, A., Staat M. (2006) “Health care reforms
Observation Error”© 2002 Journal of the
in Ukraine: The efficiency of hospitals and polyclinics.
American Statistical.
36
Sheiman, I. (1993), “New methods of finance and
Sengupta, J., K., (1998) “Testing allocative efficiency
management in health care In Russian Federation”,
by data envelopment analysis” Applied Economic
material of Health Care Reform in Developing
Letters, 1998, 5, 689-692
Countries
Conference,
Durham,
New
Hampshire, 10-13 September.
WHO (2001), “WHO European Health for all
Database. Copenhagen: WHO Regional Office
Simar, L., Wilson P. (2000), “A general
for Europe.
methodology for bootstrapping in non-parametric frontier
Wilson
models” Journal of Applied Statistics, vol. 27,
Deterministic Nonparametric Frontier Model with
No. 6, 779-802
Multiple Outputs” Journal of Business and
P.,
(1993)
“Detecting
Outliers
in
economic Statistics, vol. 11, 3
Simar, L., (2002) “Detecting Outliers in Frontier
Models:
A
Simple
Approach”
Journal
of
Wilson
P.,
(1995)
“Detecting
Influential
Productivity Analysis vol. 20, No 3, pp. 391-
Observations in Data envelopment Analysis” Journal
424
of Productivity Analysis, 6, pp.27-45
Simar, L., Zelenyuk V.(2005) Statistical inference
for aggregates of Farrell-type Efficiencies, Journal of
Worthington A., (2004) “Frontier Efficiency
Applied Economics
Measurement in Health Care: A Review of Empirical
Techniques and Selected Applications”, Medical Care
Simar, L., Wilson P. (2007), “Estimation and
Research and Review, vol. 61 No. 2, 135-170
inference in two-stage, semi-parametric models of
production processes” Journal of econometrics 136
pp. 31-64
37
APPENDIX
APPENDIX A. DESCRIPTION OF INPUT AND OUTPUT VARIABLES
Table A1: Description of input and output variables
2001
2002
2003
2004
2005
493,56
252,99
501,22
264,75
498,22
265,04
513,50
280,85
504,50
266,15
563,06
259,19
563,06
262,62
563,06
255,74
570,00
274,69
565,61
269,80
0,89
0,24
0,90
0,23
0,89
0,24
0,91
0,24
0,91
0,25
1046,44
694,65
1124,43
700,95
1204,51
753,50
1179,99
739,28
1234,04
821,23
309,14
27,19
312,26
26,14
321,66
27,76
316,94
25,03
318,08
24,45
16739,33
8739,54
16986,56
9063,46
17385,67
8990,11
17988,28
9800,68
18508,33
10219,56
2327,33
987,62
2661,39
1244,58
2960,57
1287,40
3212,12
1723,35
3701,83
2282,83
7589,11
5795,18
11137,67
12381,58
7904,99
5293,72
9007,59
8574,85
9030,01
7336,34
858,13
887,66
864,53
848,74
848,60
826,91
807,27
734,34
837,13
789,99
10,86
1,86
10,83
1,77
10,85
1,71
10,61
1,75
10,46
1,79
Medical staff
Mean
St Div
Beds in use
Output
Input
Mean
St Div
Staff per bed ration
Mean
St Div
Days of temporal disability
Mean
St Div
Working period of bed per year
Mean
St Div
Patients discharged
Mean
St Div
X-ray tests
Mean
St Div
Ultrasonic tests
Mean
St Div
Number of acute surgery
Mean
St Div
Length of treatment
Mean
St Div
38
Appendix B: Output orientation
Table B1: Results of Model 1
Model1
Bias corrected
estimator
90% Confidence
interval
95% Confidence
interval
99% Confidence
interval
0,032
0,024
0,019
Lower
bound
1,030
1,021
1,034
Upper
bound
1,138
1,099
1,094
Lower
bound
1,024
1,018
1,026
Upper
bound
1,145
1,104
1,101
Lower
bound
1,009
1,016
1,022
Upper
bound
1,157
1,117
1,109
1,098
1,072
1,083
0,039
0,031
0,025
1,038
1,026
1,042
1,160
1,127
1,123
1,032
1,023
1,036
1,173
1,134
1,132
1,010
1,019
1,028
1,187
1,150
1,136
1,023
1,025
0,037
0,045
0,967
0,957
1,081
1,099
0,958
0,947
1,090
1,108
0,935
0,923
1,109
1,128
Score
StDiv
AgEffb1
AgEffb2
EntAgEf
1,081
1,057
1,066
meaneffb1
meaneffb2
OverMean
RD_AgEf
RD_mean
Table B2: Results of Model 2
Model2
Bias corrected
estimator
90% Confidence
interval
95% Confidence
interval
99% Confidence
interval
0,043
0,029
0,026
Lower
bound
1,043
1,019
1,035
Upper
bound
1,185
1,118
1,124
Lower
bound
1,035
1,018
1,033
Upper
bound
1,194
1,123
1,131
Lower
bound
1,023
1,013
1,027
Upper
bound
1,227
1,129
1,143
1,136
1,089
1,109
0,050
0,037
0,034
1,056
1,022
1,043
1,210
1,145
1,157
1,046
1,019
1,039
1,219
1,150
1,162
1,034
1,013
1,031
1,280
1,162
1,176
1,038
1,044
0,045
0,050
0,967
0,968
1,114
1,122
0,953
0,956
1,120
1,139
0,939
0,943
1,152
1,156
Score
StDiv
AgEffb1
AgEffb2
EntAgEf
1,113
1,072
1,087
meaneffb1
meaneffb2
OverMean
RD_AgEf
RD_mean
Table B3: Results of Model 3
Model3
Bias corrected
estimator
90% Confidence
interval
95% Confidence
interval
99% Confidence
interval
0,083
0,060
0,051
Lower
bound
1,050
1,062
1,075
Upper
bound
1,317
1,264
1,234
Lower
bound
1,039
1,050
1,070
Upper
bound
1,357
1,276
1,254
Lower
bound
1,028
1,035
1,050
Upper
bound
1,419
1,313
1,296
1,221
1,209
1,214
0,104
0,089
0,075
1,066
1,073
1,099
1,424
1,380
1,351
1,053
1,063
1,092
1,439
1,387
1,382
1,041
1,038
1,072
1,483
1,473
1,416
1,024
1,013
0,086
0,099
0,895
0,859
1,165
1,160
0,876
0,824
1,208
1,225
0,863
0,804
1,259
1,271
Score
StDiv
AgEffb1
AgEffb2
EntAgEf
1,172
1,147
1,154
meaneffb1
meaneffb2
OverMean
RD_AgEf
RD_mean
39
Table B4: Results of Model 4
Model4
Bias corrected
estimator
90% Confidence
interval
95% Confidence
interval
99% Confidence
interval
0,042
0,049
0,031
Lower
bound
1,021
1,057
1,057
Upper
bound
1,148
1,211
1,164
Lower
bound
1,011
1,054
1,053
Upper
bound
1,162
1,239
1,169
Lower
bound
1,005
1,049
1,036
Upper
bound
1,220
1,285
1,188
1,088
1,182
1,143
0,044
0,077
0,049
1,022
1,076
1,070
1,165
1,313
1,231
1,018
1,066
1,057
1,177
1,343
1,257
1,006
1,059
1,048
1,210
1,384
1,270
0,964
0,925
0,056
0,068
0,876
0,813
1,051
1,023
0,851
0,797
1,061
1,056
0,831
0,780
1,077
1,070
Score
StDiv
AgEffb1
AgEffb2
EntAgEf
1,081
1,123
1,103
meaneffb1
meaneffb2
OverMean
RD_AgEf
RD_mean
Table B5: Results of Model 5
Model5
Bias corrected
estimator
90% Confidence
interval
95% Confidence
interval
99% Confidence
interval
0,038
0,029
0,023
Lower
bound
1,045
1,057
1,060
Upper
bound
1,160
1,145
1,136
Lower
bound
1,037
1,053
1,054
Upper
bound
1,178
1,154
1,148
Lower
bound
1,032
1,030
1,048
Upper
bound
1,199
1,172
1,166
1,116
1,132
1,125
0,044
0,035
0,028
1,054
1,074
1,076
1,190
1,185
1,173
1,048
1,067
1,066
1,210
1,196
1,181
1,043
1,060
1,063
1,230
1,226
1,197
0,994
0,988
0,044
0,049
0,926
0,914
1,076
1,077
0,917
0,906
1,096
1,093
0,906
0,896
1,117
1,114
Score
StDiv
AgEffb1
AgEffb2
EntAgEf
1,095
1,103
1,099
meaneffb1
meaneffb2
OverMean
RD_AgEf
RD_mean
Table B6: Results of Model 6
Model6
Bias corrected
estimator
90% Confidence
interval
95% Confidence
interval
99% Confidence
interval
0,038
0,038
0,026
Lower
bound
1,042
1,079
1,075
Upper
bound
1,163
1,198
1,157
Lower
bound
1,033
1,074
1,070
Upper
bound
1,182
1,208
1,163
Lower
bound
1,023
1,054
1,066
Upper
bound
1,215
1,245
1,189
1,125
1,159
1,145
0,043
0,046
0,031
1,056
1,094
1,091
1,192
1,235
1,194
1,041
1,090
1,087
1,202
1,251
1,203
1,026
1,068
1,077
1,248
1,279
1,224
0,972
0,972
0,049
0,054
0,894
0,881
1,050
1,052
0,877
0,866
1,062
1,077
0,861
0,838
1,117
1,095
Score
StDiv
AgEffb1
AgEffb2
EntAgEf
1,099
1,133
1,117
meaneffb1
meaneffb2
OverMean
RD_AgEf
RD_mean
40
Table B7: Results of Model 7
Model7
Bias corrected
estimator
90% Confidence
interval
95% Confidence
interval
99% Confidence
interval
0,053
0,051
0,041
Lower
bound
1,056
1,114
1,102
Upper
bound
1,227
1,275
1,226
Lower
bound
1,045
1,097
1,091
Upper
bound
1,245
1,294
1,242
Lower
bound
1,037
1,077
1,080
Upper
bound
1,281
1,320
1,273
1,168
1,226
1,201
0,074
0,068
0,053
1,058
1,126
1,114
1,296
1,346
1,288
1,046
1,107
1,104
1,310
1,360
1,298
1,039
1,079
1,087
1,354
1,420
1,331
0,957
0,954
0,053
0,073
0,870
0,830
1,049
1,086
0,863
0,814
1,063
1,106
0,843
0,804
1,077
1,121
Score
StDiv
AgEffb1
AgEffb2
EntAgEf
1,139
1,193
1,169
meaneffb1
meaneffb2
OverMean
RD_AgEf
RD_mean
Appendix C: Input orientation
Table C1: Results of Model 1
Model1
Bias corrected
estimator
90% Confidence
interval
95% Confidence
interval
99% Confidence
interval
0,039
0,031
0,024
Lower
bound
0,841
0,880
0,887
Upper
bound
0,967
0,981
0,964
Lower
bound
0,831
0,868
0,879
Upper
bound
0,977
0,985
0,969
Lower
bound
0,826
0,859
0,876
Upper
bound
0,983
0,988
0,973
0,926
0,944
0,936
0,027
0,023
0,018
0,879
0,902
0,908
0,970
0,980
0,967
0,877
0,897
0,907
0,973
0,983
0,969
0,868
0,893
0,899
0,983
0,988
0,976
0,975
0,982
0,053
0,036
0,892
0,924
1,066
1,043
0,877
0,919
1,086
1,061
0,869
0,912
1,104
1,072
Mean
StDiv
AgEffb1
AgEffb2
EntAgEf
0,910
0,934
0,924
meaneffb1
meaneffb2
OverMean
RD_AgEf
RD_mean
Table C2: Results of Model 2
Model2
Bias corrected
estimator
90% Confidence
interval
95% Confidence
interval
99% Confidence
interval
0,046
0,036
0,032
Lower
bound
0,806
0,852
0,849
Upper
bound
0,961
0,970
0,958
Lower
bound
0,798
0,846
0,846
Upper
bound
0,964
0,978
0,964
Lower
bound
0,780
0,827
0,839
Upper
bound
0,979
0,989
0,970
0,905
0,927
0,918
0,036
0,027
0,024
0,843
0,884
0,879
0,958
0,969
0,957
0,835
0,881
0,874
0,963
0,975
0,961
0,815
0,862
0,864
0,980
0,989
0,967
0,974
0,976
0,055
0,042
0,893
0,912
1,067
1,042
0,869
0,901
1,085
1,055
0,846
0,874
1,095
1,059
Score
StDiv
AgEffb1
AgEffb2
EntAgEf
0,890
0,915
0,904
meaneffb1
meaneffb2
OverMean
RD_AgEf
RD_mean
41
Table C3: Results of Model 3
Model3
Bias corrected
estimator
90% Confidence
interval
95% Confidence
interval
99% Confidence
interval
0,023
0,021
0,019
Lower
bound
0,913
0,903
0,911
Upper
bound
0,982
0,972
0,973
Lower
bound
0,902
0,899
0,905
Upper
bound
0,987
0,974
0,975
Lower
bound
0,881
0,887
0,902
Upper
bound
0,996
0,977
0,977
0,950
0,939
0,944
0,022
0,023
0,020
0,914
0,897
0,910
0,983
0,972
0,972
0,908
0,895
0,908
0,986
0,974
0,975
0,885
0,873
0,898
0,996
0,981
0,977
1,009
1,013
0,026
0,024
0,971
0,978
1,044
1,048
0,963
0,965
1,060
1,057
0,956
0,963
1,069
1,081
Score
StDiv
AgEffb1
AgEffb2
EntAgEf
0,949
0,941
0,944
meaneffb1
meaneffb2
OverMean
RD_AgEf
RD_mean
Table C4: Results of Model 4
Model4
Bias corrected
estimator
90% Confidence
interval
95% Confidence
interval
99% Confidence
interval
0,020
0,018
0,014
Lower
bound
0,920
0,913
0,927
Upper
bound
0,987
0,971
0,971
Lower
bound
0,915
0,908
0,923
Upper
bound
0,990
0,978
0,976
Lower
bound
0,910
0,893
0,916
Upper
bound
0,998
0,985
0,981
0,960
0,946
0,952
0,019
0,018
0,013
0,926
0,915
0,929
0,987
0,971
0,973
0,922
0,908
0,925
0,992
0,978
0,975
0,913
0,897
0,917
0,997
0,984
0,981
1,013
1,015
0,029
0,027
0,965
0,972
1,058
1,065
0,957
0,964
1,070
1,069
0,954
0,958
1,082
1,078
Score
StDiv
AgEffb1
AgEffb2
EntAgEf
0,958
0,946
0,951
meaneffb1
meaneffb2
OverMean
RD_AgEf
RD_mean
Table C5: Results of Model 5
Model5
Bias corrected
estimator
90% Confidence
interval
95% Confidence
interval
99% Confidence
interval
0,046
0,039
0,030
Lower
bound
0,773
0,771
0,788
Upper
bound
0,926
0,893
0,889
Lower
bound
0,771
0,761
0,782
Upper
bound
0,933
0,897
0,898
Lower
bound
0,755
0,738
0,770
Upper
bound
0,945
0,924
0,921
0,864
0,847
0,855
0,039
0,034
0,026
0,802
0,789
0,810
0,927
0,895
0,894
0,787
0,782
0,799
0,931
0,906
0,904
0,777
0,768
0,787
0,947
0,921
0,923
1,021
1,022
0,071
0,061
0,912
0,921
1,141
1,119
0,899
0,915
1,156
1,146
0,886
0,906
1,176
1,166
Score
StDiv
AgEffb1
AgEffb2
EntAgEf
0,853
0,837
0,844
meaneffb1
meaneffb2
OverMean
RD_AgEf
RD_mean
42
Table C6: Results of Model 6
Model6
Bias corrected
estimator
90% Confidence
interval
95% Confidence
interval
99% Confidence
interval
0,047
0,038
0,026
Lower
bound
0,780
0,759
0,799
Upper
bound
0,938
0,882
0,878
Lower
bound
0,772
0,741
0,783
Upper
bound
0,947
0,890
0,884
Lower
bound
0,738
0,721
0,769
Upper
bound
0,969
0,918
0,920
0,867
0,833
0,847
0,043
0,034
0,024
0,796
0,780
0,809
0,934
0,886
0,883
0,784
0,761
0,801
0,946
0,899
0,893
0,752
0,755
0,787
0,966
0,913
0,919
1,051
1,043
0,084
0,075
0,918
0,927
1,194
1,176
0,903
0,903
1,204
1,180
0,887
0,884
1,257
1,208
Score
StDiv
AgEffb1
AgEffb2
EntAgEf
0,861
0,821
0,838
meaneffb1
meaneffb2
OverMean
RD_AgEf
RD_mean
Table C7: Results of Model 7
Model7
Bias corrected
estimator
90% Confidence
interval
95% Confidence
interval
99% Confidence
interval
0,027
0,026
0,020
Lower
bound
0,856
0,832
0,853
Upper
bound
0,944
0,914
0,914
Lower
bound
0,848
0,821
0,845
Upper
bound
0,951
0,925
0,922
Lower
bound
0,839
0,811
0,832
Upper
bound
0,965
0,937
0,932
0,918
0,886
0,900
0,023
0,024
0,017
0,880
0,842
0,869
0,952
0,927
0,928
0,870
0,830
0,864
0,957
0,931
0,932
0,862
0,826
0,853
0,970
0,941
0,940
1,033
1,037
0,042
0,037
0,966
0,974
1,103
1,099
0,961
0,966
1,114
1,114
0,946
0,959
1,123
1,121
Score
StDiv
AgEffb1
AgEffb2
EntAgEf
0,903
0,874
0,885
meaneffb1
meaneffb2
OverMean
RD_AgEf
RD_mean
43