Evaluating academic activities using DEA
Raffaele Pesenti, Walter Ukovich
Dipartimento di Elettrotecnica, Elettronica ed Informatica,
Università di Trieste
tel.: +39 40 676 7134/5,
e.mail: pesenti/[email protected]
Abstract
The paper uses Data Envelopment Analysis (DEA) to assess the efficiency of the Depertments of the University of Trieste, considering both research and teaching issues. General
guidelines for selecting input and output flows are proposed, different DEA models are considered and discussed, and the quantitative outcomes of the evaluation are analyzed.
Contents
1 DEA models
2 The
2.1
2.2
2.3
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3
3
4
4
5
6
6
7
7
8
9
3 Evaluation of the departments of University of Trieste
3.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.1 Efficiency ratings . . . . . . . . . . . . . . . . . . .
3.2.2 Virtual weights . . . . . . . . . . . . . . . . . . . .
3.2.3 Targets . . . . . . . . . . . . . . . . . . . . . . . .
3.2.4 Resource allocation . . . . . . . . . . . . . . . . . .
3.2.5 Nationwide comparisons . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
9
9
10
10
12
12
14
15
2.4
2.5
modelling process
DMUs . . . . . . . . . . . . . . . . . . . .
General principles for selecting inputs and
Inputs . . . . . . . . . . . . . . . . . . . .
2.3.1 Human resources . . . . . . . . . .
2.3.2 Funds . . . . . . . . . . . . . . . .
Outputs . . . . . . . . . . . . . . . . . . .
2.4.1 Teaching . . . . . . . . . . . . . .
2.4.2 Research . . . . . . . . . . . . . . .
2.4.3 Fund raising . . . . . . . . . . . .
Choice of the DEA model . . . . . . . . .
2
. . . . .
outputs
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Introduction
At the beginning of 1996, an experimental project was started by the University of Trieste, Italy,
devoted to assess how Data Envelopment Analysis (DEA) models could be used to evaluate academic activities.
The purpose of the project was threefold:
• to understand how (and whether) DEA could be useful in assessing efficiency of academic
activities;
• to devise practical guidelines for using DEA in an academic environment;
• to derive practical insights on the policy of the University of Trieste from DEA evaluations.
The scope of the present paper is not only to describe the results of this project, but also to
report on the way it has been considered by the academic community and on the impact it had
upon it.
In general terms, the purpose of the project is essentially methodological, in the sense that it
seeks a sound methodology to evaluate academic activities. Nevertheless, it is carried out on the
base of a practical evaluation activity, using real data.
In this sense, the aim of the present paper is to derive results of general validity from a practical
case, in order to make statements that are applicable, or at least meaningful, beyond the specific
situation of the University of Trieste, or even of the Italian Universities.
Although the DEA literature is very broad, only few papers deal with evaluation issues in an
academic environment.
1
DEA models
DEA models provide different ways to assess the relative efficiency of several Decision Making
Units (DMUs) which use resources of the same types (inputs) to produce results of the same types
(outputs).
The basic problem of finding coherent weights to compare non homogeneous input and output
quantities (“comparing apples with oranges”), and of gaining the consensus of all DMUs at the
same time, was brilliantly solved by Charnes, Cooper and Rhodes in 1978 [6] by letting each DMU
to determine in turn the most favorable weights for its own efficiency, and by accepting the resulting
efficiency assessment. The resulting optimization model (called the DEA–CCR model) turns out
to be
max hj0


K
X

wk yj0 k 


 k=1

=  I

X


vx 
hj0 ,v,w
hj0
(1a)
(1b)
i j0 i
i=1
K
X
wk yjk
k=1
I
X
≤ 1 ∀j
(1c)
≥ ε
(1d)
vi xji
i=1
wk , vi
2
∀k, i
where xij denotes the level of the ith input used by the DMU j, ykj the level of the kth output
yielded by the DMU j, w = [w1 , w2 , . . . , wK ] and v = [v1 , v2 , . . . , vI ] are weights to be decided for
the K outputs and the I inputs, respectively.
The above problem can be formulated as a Linear Programming Problem for each DMU.
The dual formulation of the linear programming problem derived from (1) is of great interest:
!
I
K
X
X
+
−
min
ϑ−ε
sk +
si
(2a)
+ −
ϑ,λj ,sk ,si
k=1
N
X
λj yjk − s+
k
= yj0 k
N
X
λj xji + s−
i
= ϑxj0 i
i=1
∀k
(2b)
j=1
∀i
(2c)
≥ 0 ∀k, i, j.
(2d)
j=1
−
λ j , s+
k , si
Each feasible solution of (2) defines a conic combination of the production plans associated to
the observable DMUs (i.e. the DMUs under consideration), such that:
• the resulting combination of the outputs is not smaller than the output of the DMU j0 ,
• the conic combination of the inputs is not as large as ϑ times the input of the same DMU.
Since 1978, several different versions of DEA models have been proposed in the literature,
showing different useful properties which make them apt to different practical situations.
A complete presentation of DEA models is far beyond the scope of the present paper. The
reader is referred to the several existing books (such as, for instance, [5], [15]) and review papers
(such as, for instance [17]).
2
The modelling process
In this section, the modelling process is described that led to formulate a DEA model to assess
efficiency at the University of Trieste. It is worth to point out that the modelling process in this case
has been particularly critical and, as a consequence, its results seem to be quite meaningful, not
only for the specific considered situation, but also in view of the aim of devising general guidelines
for using DEA in an academic environment, as discussed in the Introduction. In particular, we
discuss how DMUs, inputs and outputs have been identified.
2.1
DMUs
A natural choice for identifying DMUs was to consider the Departments of the University, as it was
done in [18]. However, the University of Trieste (like most Italian Universities) has two different
types of Departments: the more recent ones, each with an autonomous budget, and the older ones,
whose costs are covered by the general University budget.
Of course, Departments of the first kind are well amenable to DEA analysis, since the resources
they use are recorded in their budgets. However, it was decided to consider also the Departments of
the older fashion, in order to try to provide a more complete coverage of all the different academic
situations: in particular, the Medicine and Law Faculties mostly have Departments of the old
type. So it was decided to consider also the old Departments, grouped into Faculties, and see if
the analysis would show any drawback in doing so.
3
2.2
General principles for selecting inputs and outputs
In general, deciding which are the inputs and the outputs for a DEA model is not a trivial task.
In some cases, it could be questionable even deciding if some specific flow has to be considered as
an input or an output.
To overcome such a kind of drawbacks, the following principles have been formulated both on
the base of findings of the literature and of the practical experience:
P1.
It is sensible to keep the cumulative number of inputs and outputs as low as possible, in
order to enhance the discriminatory power of the model; in practice, it is suggested in [5]
that the cumulative number of inputs and outputs should not exceed one–third of the number
of considered DMUs;
P2.
Highly correlated inputs (or outputs) are redundant: all of them, except one, can be dropped,
without worsening the effectiveness of the model (cf. [19]);
P3.
Inputs which do not influence any output give evidence that the set of outputs is incomplete:
in fact, they represent used resources that do not produce any measured results; however, if
the relevant outputs cannot be measured, it is preferable to drop such inputs;
P4.
Data availability should not limit the choice of inputs and outputs: in prospect, the need for
new data can be an acceptable result of the project;
P5.
As far as the distinction between inputs and outputs is concerned, the practical rule has been
followed of considering as inputs flows whose reduction appeared to improve the performance
of a DMU, and as output flows which seemed wise to increase.
P6.
It is sensible to consider inputs and outputs that can be meaningful to the potential end users
of the models; in fact, scarce comprehensibility have been often reported in the literature as
a major drawback of DEA models; as an example, see [11].
P7.
The input and output flows considered by the model must cover all the relevant activities of
all the considered DMUs; otherwise, some DMU could be erroneously underrated, if some of
its best practices is not captured.
Of course, these principles have not to be taken as imperative regulations, but rather as general
guidelines to which one may refer to in ambiguous situations. In fact, in some cases it could be
difficult to comply with all of them simultaneously; possible conflicts should be solved taking into
account the specific situation, as it will be shown in some examples in the following sections. In
general, however, these principles turn out to be convenient in several instances, as it is documented
below. Accordingly, they could be considered as a first contribution to the methodological results
of this paper.
2.3
Inputs
In practice, the following flows have been considered as possible inputs for the model:
a) ordinary funds, i.e. funds transferred from the central University budget to the Departments,
for the following purposes:
1. for general operational activities;
2. for purchasing books;
3. for subscribing journals;
4. for operating teaching laboratories;
b) human resources:
4
1
2
3
4
5
6
7
8
ordinary funds
funds for books
funds for journals
funds for laboratories
teaching personnel (number)
non teaching employees (number)
teaching personnel (salaries)
non teaching employees (salaries)
Table 1: Flows considered as possible
inputs
2
3
4
5
6
7
8
1
.14
.54
.59
.96
.96
.91
.93
2
1.00
.06
-.07
.20
.18
.26
.09
3
4
5
6
7
1.00
.35
.52
.53
.40
.41
1.00
.50
.52
.42
.49
1.00
1.00
.86
.87
1.00
.85
.87
1.00
.97
Table 2: Correlation indices between the flows of
Table 1
5. number of teaching personnel (assistant, associate and full professors);
6. number of non–teaching employees (administrative and technical);
7. salaries of teaching personnel;
8. salaries of non–teaching employees.
For ease of reference, they are reported in Table 1.
Concerning the first four possible inputs considered, it should be pointed out that it seemed
more appropriate to consider funds, i.e. available money, rather than expenditures, as in [18], since
the capability of effectively using the available financial resources is a meaningful attitude towards
efficiency. In fact, available but not used capitals bear a not negligible relevance in the budgets of
the University of Trieste and of its Departments.
In order to select the more appropriate flows to be considered as inputs for DEA models, a
preliminary analysis has been carried out on the possible inputs considered so far. First, the
correlation factors between them have been calculated. They are shown in Table 2.
2.3.1
Human resources
It turns out that the number of teaching personnel (5) and of non–teaching employees (6) are
strictly correlated (up to more than 97%) with the respective salaries (7), (8). As a consequence,
according to the principle P2, only salaries (5), (6) have been retained as inputs, and the other
correlated flows have been dropped from the input set. The choice of retaining salaries instead of
the number of people seemed wiser since salaries give a more precise aggregate measure of personnel
skills and responsibilities.. Although neither departments nor universities have full control of the
salary levels, since they are stipulated by national contracts, nevertheless they can acquire positions
at different levels, both for teaching and non teaching personnel.
Now the question is how to consider these two inputs. They are quite strictly correlated (more
than 85%), although below the threshold level of 90% that has been adopted to exclude highly
correlated flows. However, it should be noticed that such a high level is mainly produced by
the data of the Medicine Faculty. In fact, dropping the data of Medicine, the correlation level
falls to 43%. By the way, this an argument against including the Medicine Faculty in the analysis,
according to what was as done in [18], although for different reasons. Thus two alternatives remain:
to consider them separately or to aggregate them in some way, e.g., by imposing some bounds on
their weights. The first alternative may conflict with principle P1; more important, it could allow
for the possibility of a high efficiency score by imposing a very large weight to the non teaching
personnel salaries only, which is clearly a nonsense. In fact, it is true that non teaching personnel
may have a relevant role, e.g., in bearing the burden of bureaucratic activities (and for this reason
it cannot be dropped as an input); on the other hand, it would be absurd to evaluate the efficiency
of an academic department on this base only. Then the second alternative remains of somehow
aggregating these two inputs. According to principle P1, their sum has been taken as a single
input (this is equivalent to consider them separately, with the additional constraint of having equal
5
weights). Similar solutions, such as imposing to the weight of non teaching employees salaries to
be not larger than teaching personnel salaries, give almost equivalent results; furthermore it may
appear to be less equitable.
2.3.2
Funds
According to principle P1, it was assumed to consider no more than two inputs. Therefore, some
form of aggregation is necessary for funds too. However, funds for books are distributed to the
few departments which have an autonomous library. Therefore, since we were not able to charge
to the departments the cost of the interdepartment libraries, funds for books have been dropped.
Conversely, the same effect is almost irrelevant for funds for journal, since all the DMUs do receive
these funds. Their magnitude order is comparable with the one of funds for the ordinary operations.
Concerning funds for laboratories, they are distributed to almost all the departments. However, it
was clear that considering them in the input set would penalize too much scientific departments
with relevant laboratories, since no output related to laboratories was available (see below). Thus,
according to principle P3, funds for laboratories have been excluded. Note however that a more
satisfactory alternative would have been to devise some outputs to assess the results provided by
laboratories, according to principle P4. The remaining two funds: for ordinary operations and for
journals, have been aggregated to provide the second input. Note that their correlation factor is
quite low.
It must be pointed out, however, that excluding funds for books and laboratories is not completely satisfactory. This is also due to the fact that there are some common facilities, such as the
central and some Faculty libraries, and laboratories (mostly computer laboratories), which are used
by some Department members, while the resources they use are not inputs for such Departments.
Although such centralized facilities are sensible to get scale economies, they risk to hidden the
actual distribution of the used resources.
A possible solution to such a drawback is to devise a reasonable way to redistribute (practically
or virtually) such resources among the actual users. However, any actual redistribution scheme
is not presently contemplated: it would bear the risk of disincentivate the use of such facilities;
moreover, it would imply a non negligible operational burden. Virtual redistribution schemes (i.e.
without real money transfer) would also be questionable, or complicated: the simplest possibility,
of redistributing the resources of common facilities among the Departments according to their
relative size, would just produce an input with high correlation with the first input we already
have.
In view of the above considerations, the alternative we choose, of not considering funds for
books and for laboratories, is an incentive to use the common available facilities. After all, it
should also be mentioned that considering ordinary funds and funds for journals as inputs is in
agreement with [2] and a refinement of [10] and [18]. There the inputs are the operational costs
and the salaries.
2.4
Outputs
According to [12], the relevant outputs of the activities of the academic staff are: supervision,
teaching, administration and research. In [2], only teaching and research are considered. A third
alternative is provided by the Conference of the Rectors of Italian Universities (CRUI) [8], which
surveys the academic activities in all the Italian Universities, and considers: teaching, research and
fund rising. According to principle P7, it seems wise to consider the union of the above activities as
a reasonable output set of the analysis. However, supervising activity is not yet fully implemented
in the University of Trieste, and therefore data are not yet available. According to principle P4,
this points out the need for monitoring such an activity. Concerning administrative activities, they
are certainly important, not only with respect to the internal management of departments (which
as a first approximation could be considered equivalent for all the DMUs), but mostly concerning
academic and professional appointments, positions or offices. Unfortunately, this information is
6
rather difficult to get and, so far, is not recorded. Therefore, we reduce to outputs used by CRUI:
teaching, research and fund rising.
2.4.1
Teaching
For teaching, the number of courses held by the personnel of each DMU has been considered as
a first output. This is coherent with principle P5, since a larger number of offered courses gives
more flexibility and choice alternatives to the students. On the other hand, offering more courses
requires larger teaching resources (number and duty level of teaching personnel).
The number of examinations registered in a year has been taken as a second output. It was
considered as a quantitative measure of the “delivered products”, in contrast with the level of
“offered products” given by the first output. In this way, DMUs not having a wide range of offered
courses (and thus achieving scale economies) are not penalized.
It must be pointed out that we only use quantitative outputs for teaching activities, while a
qualitative assessment would also be advisable. However, quality measures are not yet available in
the University of Trieste. Even subjective evaluations, such as the quality level of the courses as
perceived by the students, are not yet implemented. More sophisticated evaluations, such as the
impact of graduates’ abilities on the success of their professional career, could also be questionable,
at least for the possibility of attributing them to the DMUs.
2.4.2
Research
For research activities, only published papers and books have been considered. Again, this is only a
quantitative measure, and the need for qualitative assessment is pointed out, according to principle
P4. However, data relevant to the quality of research, such as in [2], are not available. Anyway,
data about fund raising could be considered as a proxy for these data.
A major problem in assessing research activities on the basis of published papers is inter–area
comparisons. As a matter of fact, some disciplines, for instance Medicine, have publication rates
much higher than other ones. This problem has not been dealt with in [18], where the number of
papers has been considered. In this paper, the problem has been solved considering the national
average number of papers per researcher Nd in each scientific area d, on the basis of the data of
CRUI [8]:
P
nk
Nd = Pk∈Kd ,
k∈Kd rk
where
Kd is the set of the departments of the scientific area d under consideration in all Italian Universities,
nk is the number of papers of the researchers of the kth department in the area d,
rk is the number of the researchers of the kth considered department.
Then, the ratio
nj0
Nd
(3)
is taken as a measure of the department research activity, where nj0 is the number of papers published by the researchers of the department j0 under consideration. Such a ratio can be interpreted
as the number of researchers that would produce the same number nj0 of papers, according to the
national average.
As regards data availability, a complete database of all papers published by the researchers of
the University of Trieste is presently considered for implementation: data are not yet available.
We considered as a proxy the number and type of papers listed in the requests for local research
funds. They have been classified in six categories:
7
1
2
3
4
5
6
international journals
international conferences
national journals
national conferences
books
other
Table 3: Categories for papers
2
3
4
5
6
1
0.57
0.22
0.48
-0.02
-0.24
2
1.00
0.24
0.69
0.16
0.08
3
4
5
1.00
0.48
0.53
0.33
1.00
0.24
0.03
1.00
0.50
Table 4: Correlation indices between the
elements of Table 3
1. papers published on international journals;
2. papers presented at international conferences;
3. papers published on national journals;
4. papers presented at national conferences;
5. books;
6. other (internal reports, and similar).
For ease of reference, they are reported in Table 3. For each of these categories, the ratio
corresponding to Eq. (3) has been calculated. Then, they have been aggregated in two groups:
the first contains papers appeared in international journals and conferences (categories 1 and 2
of Table 3 — for ease of presentation, we shall refer to it as international papers), the second as
all the remaining papers (categories 3 ÷ 6 of Table 3: papers appeared in national journals and
conferences, books and other — for ease of presentation, we shall refer to it as other papers).
For the sake of completeness, the correlation factors have been also computed and are shown in
Table 4. They substantially confirm the goodness of the choice of our inputs, since, as an example,
the international journal papers and the international conferences are not significantly correlated
between each other (57%), whereas both of them are, obviously, more strictly correlated with their
sum: 92% and 85%, respectively.
Concerning the relative importance of international papers and other papers, it is clear that it
may have different relevance in different disciplines, since international papers are fundamental in
non humanistic areas, whereas the converse could be true in some humanistic areas. To overcome
this drawback, categorization has been proposed in the literature [7]. It consists in introducing
an additional binary input to discriminate between humanistic and not humanistic departments.
In practice, this allows the humanistic departments not to consider the non humanistic departments for comparison, just by giving a suitably high weight to this new input. However, such an
approach seems to be a too restrictive one. Instead, an alternative original method has been considered: restricting the weight of other publications to be not larger than the weight of international
publications, but only when non humanistic departments are under evaluation.
2.4.3
Fund raising
All sources of funds for research have been considered. In Italy they typically come from the Ministry of University, both directly and through the National Research Council, from other agencies
(as the Italian Space Agency), from industries, and from the European Community. Clearly, the
relative importance of the different funding sources depends on the specific research area. To get a
common basis for comparison, they have been first aggregated and then normalized as it has been
done for the data of research papers in the Section 2.4.2. In this case however, the aggregation step
has been performed before normalization, since the quantities under concern are homogeneous.
8
2.5
Choice of the DEA model
Among the several dozens of DEA models available in the literature [22], it was decided to considere
just the fundamental ones. A first meaningful property discriminating among the different models
is how they deal with returns to scale. In fact, the CCR model assumes constant returns to scale,
whereas, as an example, BCC models allow for variable returns to scale. So a first question was to
decide if variable returns to scale are appropriate for academic activities.
In the authors’ opinion, variable returns to scale should not be considered for teaching outputs,
at least for two reasons: first, teaching activities, considered as a service provided to the students,
should not depend on the department size; second, the teaching effort of each professor should
not vary according to the size of the department. Similar considerations do not necessarily apply
for the research and fund rising outputs. In these cases, relevant differences in size could justify
different efficiency evaluations, at least in principle. However, it must be observed that the relative
size of the departments under consideration is rather uniform. As a consequence, variable returns
to scale may be allowed provided there is no significant decrease in the discrimination power of the
model. Accordingly, experiments have been performed using, beside CCR, BCC and FDH [21].
In the considered DEA models, i.e., CCR and BCC, efficiency can be also interpreted, using
their dual formulation (cf., for instance, 2), as the sum of two components:
ϑ − ε · S.
(4)
The first component ϑ is the scaling ratio of the appropriate combination of the DMU inputs and
outputs that produces the minimal “ghost” DMU dominating the one onder consideration. The
second component ε · S is the product of the non archimedean constant ε and the sum S of the
disposal variables, which represent the slacks of the input and output values of the DMU under
consideration with respect to the ghost values. To avoid numerical problems produced by non
archimedean constants, the two–stage approach has been used in implementation as suggested in
[13]. Its first phase finds ϑ, and the second phase produces S. Being the value of ε arbitrarly small,
only the ϑ values are are shown in the following tables. It should be noted that this may produce
an overestimate of the efficiency of some inefficient DMUs [3].
3
Evaluation of the departments of University of Trieste
The results of the previous section have been adopted as guidelines to evaluate the departments
of University of Trieste. CCR, BCC, and FDH models have been used. For the the first two
models, both the input and output versions have been considered. In each model, as indicated
in Section 2, the weight of other papers was constrained to be not larger than the one of the
international papers, when a non humanistic department was under consideration. All models
have been implemented using GAMS [14] and solved using Cplex [9]. Owing to the limited size of
the problem (37 DMUs were involved), solving 37 linear programming problems was an affordable
task. Another implementation of DEA models using GAMS is reported in [22].
In order to rank efficient DMUs, superefficiency was allowed by not bounding to one the efficiency of the DMU under consideration, as done in [1]. In some cases, where multiple optimal
solutions have ben experienced, the one has been considered with maximum values of the minimum
weights.
3.1
Data
Table 5 shows the data used for the analysis. Ordinary funds and salaries are expressed in millions
of Italian lire, papers and research funds are expressed in relative units according to Eq. (3). Besides
the correlation already discussed in Section 2, no other a–priori analysis has been performed. For
obvious reasons, conventional names for the DMUs have been adopted.
9
DMUs
D01
D02
D03
D04
D05
D06
D07
D08
D09
D10
D11
D12
D13
D14
D15
D16
D17
D18
D19
D20
D21
D22
D23
D24
D25
D26
D27
D28
D29
D30
D31
D32
D33
D34
D35
D36
D37
Inputs
Salaries
Ord.
1176.26
2530.87
3152.71
971.19
1271.60
1375.90
3670.90
1580.38
1566.82
2953.88
1860.24
1832.09
3061.29
1306.14
2121.63
2001.45
1011.30
1497.89
913.94
2191.01
2060.66
3432.65
3154.72
1363.82
2105.04
2405.58
2167.07
1972.02
1779.38
4003.43
3178.06
4498.57
2366.14
848.13
3539.65
1476.51
12967.93
Funds
47.08
157.45
218.97
64.72
35.04
53.69
131.91
61.56
48.19
179.03
51.60
75.68
133.64
57.92
105.08
47.30
24.15
49.37
64.79
72.60
112.76
306.64
118.27
34.95
94.86
116.84
120.26
87.78
85.06
267.87
114.54
185.00
90.97
38.53
163.74
77.74
468.92
Exams
35
694
1149
1124
1435
829
1409
556
650
793
332
204
440
544
366
1029
1175
789
445
1953
573
729
709
1086
228
2317
654
675
589
2186
6264
3147
844
1602
5020
878
2693
Courses
6
26
30
17
19
37
49
29
17
23
13
22
23
16
26
26
46
16
11
34
19
37
35
20
12
26
25
59
19
82
60
194
25
11
32
24
93
Outputs
Intl. Paps.
Other Paps.
12.33
21.23
32.15
16.50
13.26
65.53
3.39
20.32
0.00
29.00
1.00
51.17
86.63
88.00
16.82
20.92
3.00
63.00
64.59
14.88
49.05
20.40
37.23
38.40
9.53
36.65
5.52
19.52
35.46
40.82
1.00
63.83
3.00
32.00
12.08
20.67
10.81
12.26
16.28
70.33
19.03
18.01
35.87
15.90
7.00
100.50
1.00
38.17
19.67
45.36
4.00
32.00
8.94
10.30
4.00
84.50
3.74
22.85
66.28
50.58
3.00
94.00
36.50
144.50
4.00
83.00
0.00
33.33
2.00
163.24
2.00
124.00
90.48
156.79
Res. Funds
19.03
24.56
21.34
22.66
7.71
11.49
34.45
20.20
18.53
18.99
14.74
10.01
115.33
8.09
47.79
15.79
5.57
9.77
8.67
40.32
8.14
33.56
50.12
17.39
79.76
21.08
12.09
20.67
9.64
43.83
23.38
24.53
28.08
2.93
32.55
25.37
106.53
Table 5: Data used for the evaluation.
3.2
3.2.1
Results
Efficiency ratings
Table 6 shows the efficiency ratings provided by the CCR, BCC and FDH models. CCR and BCC
have been implemented with the additional constraint limiting the weight for other publications
to be not greater than the weight of the international ones. Observe that the used models show
a decreasing discriminating power: CCR has 12 superefficient DMUs, i.e., with efficiency rating
greater than one, and two other DMUs have an efficiency rating quite near to one; however, they
are not fully enveloped [3]. As a consequence, their efficiency depends on the value of the non
archimedean constant, and therefore their actual efficiency rating could be slightly smaller than
the value shown in Table 6. Note that the same cannot occur with superefficient DMUs, since their
efficiency values greater than 1 prove that they are necessary to define the efficient frontier, i.e.,
they define a proper vertex of it. BCC has 18 efficient DMUs; finally, FDH has 36 efficient DMUs.
Note that BCC and FDH do not produce superefficient DMUs, since this concept is hardly tackled
by these models.
As it was pointed out in Section 2.5, BCC allows for variable returns to scale with respect to any
10
DMUs
D01
D02
D03
D04
D05
D06
D07
D08
D09
D10
D11
D12
D13
D14
D15
D16
D17
D18
D19
D20
D21
D22
D23
D24
D25
D26
D27
D28
D29
D30
D31
D32
D33
D34
D35
D36
D37
1
CCR
69.24
67.48
47.74
110.32
83.20
77.44
112.30
80.48
93.64
88.57
144.73
86.99
104.36
47.75
103.07
92.70
200.45
60.59
72.49
111.54
48.16
61.25
76.16
99.97
121.33
61.26
41.28
89.80
37.28
98.03
129.99
111.29
68.73
105.05
93.76
179.77
46.84
2
BCC
100.00
69.46
47.74
100.00
88.62
79.65
100.00
86.02
97.81
88.58
100.00
95.98
100.00
73.84
100.00
94.20
100.00
76.52
100.00
100.00
57.25
61.68
83.42
100.00
100.00
62.07
47.88
90.26
52.96
100.00
100.00
100.00
69.62
100.00
100.00
100.00
100.00
3
FDH
100.00
100.00
93.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
DMUs
D01
D02
D03
D04
D05
D06
D07
D08
D09
D10
D11
D12
D13
D14
D15
D16
D17
D18
D19
D20
D21
D22
D23
D24
D25
D26
D27
D28
D29
D30
D31
D32
D33
D34
D35
D36
D37
Table 6: Efficiency ratings.
1
CCR
69.24
67.48
47.74
110.32
83.20
77.44
112.30
80.48
93.64
88.57
144.73
86.99
104.36
47.75
103.07
92.70
200.45
60.59
72.49
111.54
48.16
61.25
76.16
99.97
121.33
61.26
41.28
89.80
37.28
98.03
129.99
111.29
68.73
105.05
93.76
179.77
46.84
2
weights
69.24
67.48
47.74
110.32
83.68
78.09
112.30
80.48
94.40
88.57
144.73
86.99
104.36
47.75
103.07
94.90
200.45
60.59
72.49
111.54
48.16
61.25
78.33
102.89
121.33
61.26
41.28
90.08
37.28
98.03
129.99
111.29
69.89
106.02
93.97
183.18
46.84
3
salaries
79.30
67.77
49.54
110.95
83.20
192.50
117.08
87.36
101.20
89.97
180.58
87.86
136.63
53.36
106.68
96.25
200.45
134.50
78.54
130.56
51.04
61.33
126.78
99.97
146.32
65.85
43.03
99.73
37.84
110.95
158.29
119.18
75.27
106.21
94.49
181.94
47.74
Table 7: Efficiency ratings with some
constraints dropped.
output flow. This explains the general improvement of efficiency of DMUs that where inefficient
according to CCR (actually, six of them become efficient). It is interesting to notice that this
improvement is generally due to research outputs and not to teaching outputs. In fact, the hybrid
model proposed in [17] allowing for variable returns to scale for research outputs only provides
exactly the same results as BCC. This is not surprising, since it turns out that CCR shows efficient
DMUs of significantly different size, which are efficient mainly to teaching outputs, as it can be
seen in the following Table 8.
Because of the weak discriminatory power of the other models and of the concerns on the
appropriateness of assuming variable returns to scale, as mentioned in Section 2, CCR turns out to
be the most appropriate DEA model for evaluating academic departments. This is an important
methodological result of our project. It should be mentioned that CCR was used in [18], but with
neither theoretical nor empirical justification.
Table 7 shows the efficiency ratings, only for the case of the CCR model, when either the
constraint relevant to the weights of the other papers and the international papers is dropped
(column 2), or the salaries of teaching and non teaching personnel are considered separately (column
11
3). For the sake of convenience, Table 7 also shows again the ratings obtained by the original CCR
model of Table 6 (column 1). It turns out that efficiency rates with no weight constraint are
always almost identical to the constrained case. Deviations are present mainly for humanistic
departments, and they always are quite small (less than 2%). This makes practically equivalent
the two models. For reasons of convenience, the constrained model is used in the following. By
the way, this kind of results proves the effectivenes of the normalization of the research and fund
raising outputs on a nationwide basis. Cosidering separately teaching personnel and non teaching
employees produces more significant variations. In particular, six previously inefficient DMUs
become superefficient. For three of them, the efficiency rating almost doubles. Even if, as already
pointed out, the efficiency of a department cannot rest on the scarcity of non teaching employees
only, these results can be used to detect such deficiencies.
3.2.2
Virtual weights
Table 8 shows for each DMU j0 the percent virtual weights:
w? y
P k ?j0 k · 100
r wk yj0 r
for each output flow yj0 k and
v? x
P i ?j0 i · 100
s vi xj0 is
for each input flow xj0 i , where wk? and vi? are the optimal weights for DMU j0 . In fact, according to
[19], virtual inputs and outputs show exactly how each input or output takes part in the observed
efficiency rating. In other words, a DMU with particularly high virtual values for a given output
or a given input indicates that it considers itself as a possible site of best practice with respect to
the capability of using the given input or to produce the considered output.
Observing the virtual values for inputs, it turns out that only 8 DMUs have a 100% value
for ordinary funds. As most of them belong to the humanistic area, this result could suggest to
check if the scientific departments get more funds. Concerning outputs, it must be observed that
15 DMUs base their efficiency rates mainly on scientific outputs, funds and papers (their global
virtual weights are greater than 75%), whereas three only of them mainly base their efficiency rates
on teaching outputs. This seems to suggest that a major emphasis is placed on research than on
teaching activities, and this complies with the rules for selecting teaching personnel. The remaining
19 DMUs show a better balance between teaching and research activities; however, only two of
them turn out to be efficient. This is not surprising, as it is known that DEA highlights the DMUs
which are sites of best practices. In fact, it turns out that DMUs with the most balanced virtual
weights for outputs are among the most inefficient ones. This phenomenon is clearly enhanced by
the fact that superefficient DMUs are allowed, since in this case the possibility of multiple solutions
is reduced with respect to the case in which efficiency rates are saturated to one.
Further insights on this issue can be obtained using crossefficiency evaluations [10]. The relative
virtual weights obtained by limiting efficiency of the current DMU to one and maximizing as a
sD34dary objective a proxy of the other DMUs efficiencies for the efficient DMUs are shown in
Table 9 (the virtual values of non efficient DMUs do not change). It should be noted that in this
way much more balanced virtual values are obtained. In particular, only three DMUs may be
considered as “maverick” according to [10], i.e., their efficiency is mainly due to their excellence in
a restricted area. In fact, each of them has a relative virtual weight greater than 66%.
3.2.3
Targets
Table 10 shows for each DMU the targets provided by the input oriented model. Similarly,
Table 11 shows for each DMU the targets provided by the output oriented model. For the sake of
clarity, they are expressed in percentage with respect to actual input and output values. Positive
12
DMUs
D01
D02
D03
D04
D05
D06
D07
D08
D09
D10
D11
D12
D13
D14
D15
D16
D17
D18
D19
D20
D21
D22
D23
D24
D25
D26
D27
D28
D29
D30
D31
D32
D33
D34
D35
D36
D37
Salaries
59.59
100.00
100.00
100.00
0.00
100.00
100.00
79.78
0.00
100.00
0.00
100.00
0.00
100.00
100.00
0.00
0.00
69.18
100.00
39.33
100.00
100.00
7.61
0.00
69.58
96.94
100.00
100.00
100.00
100.00
38.79
100.00
0.00
100.00
100.00
100.00
37.36
Inputs
Ord. Funds
40.41
0.00
0.00
0.00
100.00
0.00
0.00
20.22
100.00
0.00
100.00
0.00
100.00
0.00
0.00
100.00
100.00
30.82
0.00
60.67
0.00
0.00
92.39
100.00
30.42
3.06
0.00
0.00
0.00
0.00
61.21
0.00
100.00
0.00
0.00
0.00
62.64
Exams
0.00
10.17
22.33
33.15
65.28
0.00
10.66
0.00
0.00
14.20
0.00
0.00
0.00
21.87
0.00
8.89
26.23
32.18
23.40
24.41
23.03
7.18
0.00
42.91
0.00
66.33
18.42
0.00
22.26
13.93
86.37
0.00
0.00
65.68
54.30
0.00
2.50
Courses
0.00
18.02
15.61
17.26
0.00
55.71
7.61
37.36
3.59
0.00
0.00
23.51
6.42
22.79
19.05
0.00
73.77
11.45
16.06
0.00
18.81
22.82
5.28
2.33
0.00
0.00
32.95
53.45
25.45
24.74
0.00
77.34
3.81
0.00
0.00
0.00
8.48
Outputs
Intl. Paps.
Other
36.80
46.15
19.65
1.31
0.00
0.43
64.73
33.12
2.96
85.80
100.00
76.49
0.00
19.48
42.75
1.07
0.00
34.11
41.55
14.53
58.16
42.24
3.20
0.00
17.92
7.57
24.50
1.04
12.42
41.40
0.00
22.66
2.84
0.00
0.45
1.42
31.09
Paps.
25.30
0.00
21.58
0.00
9.42
22.03
17.00
0.00
62.02
0.00
0.00
0.00
0.00
14.83
0.00
67.89
0.00
0.00
0.00
27.16
0.00
0.00
46.03
0.05
13.59
0.00
0.00
21.89
16.33
0.00
0.00
0.00
58.97
34.32
36.35
88.04
24.88
Res. Funds
37.90
25.67
20.84
48.27
25.30
21.83
0.00
29.52
31.44
0.00
0.00
0.00
93.58
21.03
38.21
22.16
0.00
22.25
19.00
33.90
0.00
27.76
45.48
54.71
68.49
26.10
24.13
23.62
23.55
19.93
13.63
0.00
34.38
0.00
8.89
10.54
33.06
Table 8: Relative virtual weights.
values indicate that the corrisponding flows should be raised by the indicated quantity, whereas
negative values indicate that the corrisponding flows should be decreased. As a consequence,
they would reduce the efficiency score of superefficient DMUs. This explains negative values for
superefficient DMUs in the output table, and positive values in the input table. Hence, only the
targets relative to the inefficient DMUs have a practical meaning.
However, note that some negative values may appear as output target for non efficient DMU
for other papers, due to the constraints imposing that their weight must be not greater than the
one of international papers. In this case, the target for the international papers must have a
positive greater value in absolute terms. Note that targets always denote input or output values,
respectively, which would produce the component ϑ of the efficiency to be equal to one for the
considered DMU. Observe that the condiditon ϑ = 1 does not guarantee efficiency as the sD34d
component ε · S of Eq. 4 may be greater than zero. Note however that in this situation, a further
non archimedean reduction of inputs or increase of outputs makes properly efficient the considered
DMU.
Input and output targets may have different uses, depending on the decision structure of the
system under consideration. In particular, input targets could be used by a central authority
13
DMUs
D04
D07
D11
D13
D15
D17
D20
D25
D31
D32
D34
D36
Salaries
100.00
100.00
72.71
84.21
100.00
100.00
96.77
100.00
96.39
100.00
100.00
100.00
Inputs
Ord. Funds
0.00
0.00
27.29
15.79
0.00
0.00
3.23
0.00
3.61
0.00
0.00
0.00
Exams
29.01
9.62
6.73
4.55
4.31
29.13
25.86
2.72
2.75
13.77
67.36
14.91
Courses
15.55
11.86
4.96
6.94
14.51
40.41
11.71
5.07
32.15
43.83
0.00
14.44
Outputs
Intl. Paps.
Other
7.68
51.94
70.77
1.41
40.97
6.53
16.32
20.57
4.70
17.89
0.00
2.98
Paps.
9.92
11.36
0.00
0.00
0.00
15.00
15.66
10.21
32.77
15.75
30.72
39.81
Res. Funds
37.83
15.22
17.54
87.10
40.21
8.93
30.44
61.44
27.62
8.76
1.93
27.86
Table 9: Relative virtual weights by crosseficiency for efficient DMUs.
in order to reduce the resources allocated to inefficient DMUs in order to make them efficient,
provided they can mantain their output levels. Observe that this requires a technology change,
i.e., processes producing outputs from inputs must be changed. Conversely, output targets could
be used by local decision makers in order to reach efficiency by raising their outputs with the same
amount of resources. This is a consequence of the fact that in the system under consideration,
inputs are determined by the central authority, whereas outputs are, at least partially, under the
control of the DMUs. Of course, raising outputs only or dropping inputs only are just two extreme
strategies that could be mixed in any appropriate rate. Such intermediate approaches can be
studied according to other models [20]. Dropping resources in the amount indicated by the input
oriented target makes efficient inefficient DMUs if the resources gained by this reduction are not
distributed.
3.2.4
Resource allocation
The problem of redistributing resources can be approached from two different points of view.
According to the first approach, as pointed out in the previous section, inputs of non efficient
DMUs are dropped according to their targets, imposing, at the same time, that outputs levels
are unchanged. The savings in the inputs are then redistributed to the efficient DMUs. Different
possible alternatives for redistributing the resources make the problem rather complicated [4].
Alternatively, a different optimization model could be used. It does not require technology changes,
and therefore it allocates all resources to few efficient DMUs, in such way that the overall efficiency
is maximized. This can be accomplished solving the following linear programming problem [17]:
!
K
I
X
X
+
−
sk +
si
(5a)
max ϕ + ε
λj ,s+
,s−
,ϕ
i
k
N
X
λj yjk − s+
k
N
X
λj xji + s−
i
i=1
k=1
= ϕ
N
X
yjk
∀k
(5b)
j=1
j=1
=
j=1
N
X
xji
∀i
(5c)
j=1
−
λ j , s+
k , si
≥ 0 ∀k, i, j.
(5d)
The above problem identifies the maximum output that can be produced by using an amount
of resources not larger than the current one and λj xji determines the amount of the ith resource
14
DMUs
D01
D02
D03
D04
D05
D06
D07
D08
D09
D10
D11
D12
D13
D14
D15
D16
D17
D18
D19
D20
D21
D22
D23
D24
D25
D26
D27
D28
D29
D30
D31
D32
D33
D34
D35
D36
D37
Salaries
-31
-33
-52
43
-31
-23
12
-20
-33
-11
67
-13
3
-52
3
-41
61
-39
-28
12
-52
-39
-24
-22
21
-39
-59
-10
-63
-2
30
11
-34
5
-6
152
-53
Ord. Funds
-31
-60
-68
-3
-17
-34
0
-19
-6
-52
100
-24
4
-55
-16
-7
142
-39
-55
12
-70
-70
-24
0
21
-39
-76
-30
-69
-51
30
-28
-31
-12
-13
95
-53
DMUs
D01
D02
D03
D04
D05
D06
D07
D08
D09
D10
D11
D12
D13
D14
D15
D16
D17
D18
D19
D20
D21
D22
D23
D24
D25
D26
D27
D28
D29
D30
D31
D32
D33
D34
D35
D36
D37
Exams
658
48
109
59
20
55
-11
33
28
13
128
249
3
109
63
8
-6
65
38
-10
108
63
230
0
71
63
142
164
168
2
-23
43
113
-5
7
41
113
Table 10: Input targets
Courses
66
48
109
59
34
29
-11
24
7
39
110
15
-4
109
-3
28
-6
65
38
78
108
63
31
0
66
66
142
11
168
2
36
-10
46
43
88
59
113
Intl. Paps.
44
48
109
59
inf.
297
-11
24
60
13
32
15
188
110
-3
353
77
65
38
-10
108
63
206
535
-18
63
142
47
169
2
95
-10
202
8340
60
1713
113
Other Paps.
44
270
109
80
11
24
-11
99
4
274
131
15
81
109
22
2
4
91
76
-10
192
389
19
-14
-18
94
474
10
168
103
21
-6
38
-7
6
6
113
Res. Funds
44
48
109
59
20
29
-8
24
7
37
54
71
-4
110
-3
8
8
65
38
-10
119
63
31
0
-18
63
142
11
168
2
-23
16
45
148
7
33
113
Table 11: Output targets
that must be assigned to the DMU j. It is also trivial to verify that the corresponding primal
formulation minimizes the aggregate efficiency, subject to the constraints that the efficiency of all
the single DMUs should not exceed one.
Solving problem (5), it turns out that all resources should be given to 5 DMUs only. Table 12
shows for each of such DMUs the input required and the output produced in this case, together
with the new aggregate values and the current ones. Observe that these DMUs are quite well
distributed among the different scientific areas, and that they should be able to produce significant
improvements with respect to all the outputs. Of course, such a drastic solution could hardly
be viable. In order to obtain more realistic situations, appropriate constraints or cost should be
introduced in the model, as in [17].
The input oriented version of problem (5) provides the same DMUs, which are capable to yield
the current outputs, using as much as 30% less than the present resources, cf. Table 13.
3.2.5
Nationwide comparisons
Finally, the problem has been considered of assessing the nationwide validity of the efficiency scores
obtained so far. In particular, a non efficient DMU at local level could turn out to be efficient if
compared with other DMUs of other universities in the same scientific area. This could be the
15
DMUs
D04
D07
D17
D25
D36
new values
current values
Inputs
Salaries
Ord. Funds
23940,50
1594,90
34013,32
1222,14
15673,06
375,05
8273,42
372,98
9466,57
498,17
91366,87
4063,24
91366,87
4164,60
Exams
27707,37
13055,30
18210,06
896,11
5629,25
65498,08
46145,00
Courses
419,06
454,02
712,90
47,16
153,87
1787,02
1259,00
Outputs
Intl. Paps.
Other Paps.
83,57
500,90
802,68
815,38
46,49
495,93
77,31
178,28
12,82
795,02
1022,87
2785,51
720,64
1962,46
Res. Funds
558,58
319,20
86,32
313,48
162,66
1440,25
1014,69
Table 12: Output oriented aggregate efficiency.
new values
current values
Inputs
Salaries
Ord. Funds
64370,19
2862,65
91366,87
4164,60
Exams
46145,00
46145,00
Courses
1259,00
1259,00
Outputs
Intl. Paps.
Other Paps.
720,64
1962,46
720,64
1962,46
Res. Funds
1014,69
1014,69
Table 13: Input oriented aggregate efficiency.
effect of locally having a group of DMUs which are sites of best practices at national level, which
would overpenalize the other DMUs. To this end, a new set of experiments has been performed
comparing some departments of the Trieste University with all the departments of the same area
in all Italian universities, using the data of CRUI [8]. As an example, an efficient DMU and a
non efficient one have been considered. It turns out that the efficiency of the non efficient DMU
slightly increases from the local to the national evaluation, whereas the efficiency of the efficient
DMU drastically decreases. However, several issues in this problem deserve further depeening.
References
[1] P. Andersen and N. C. Petersen, “A procedure for ranking efficient units in in Data Envelopment
Analysis”, Management Science, vol. 39, No. 10, pp. 1261-1264, 1993.
[2] J.E. Beasley, “Determining teaching and research efficiencies”, Journal of the Operational
Research Society, vol. 46, pp. 441-452, 1995.
[3] A. M. Bessent, E. W. Bessent, J. Elam, and T. Clark, “Efficient frontier determination by
constrained facet analysis”, Operations Research, vol. 36, No. 5, pp. 785-796, 1988.
[4] A. Boussofiane, R. G. Dyson and E. Thanassoulis, “Applied data envelopment analysis”,
European Journal of Operational Research, vol. 52, pp. 1-15, 1991.
[5] A. Charnes, W. W. Cooper, A. Y. Lewin, and L. M. Seiford (Eds.), “Data envelopment analysis:
theory, methodology, and applications”, Kluwer Academic Publisher, Norwell, MA, 1994.
[6] A. Charnes, W. W. Cooper, and E. Rhodes, “Measuring the efficiency of decision making
units”, European Journal of Operational Research, vol. 2, pp. 429-444, 1978.
[7] W. D. Cook, M. Kress, and L. M. Seiford, “On the use of ordinal data in Data Envelopment
Analysis”, Journal of the Operational Research Society, vol. 46, pp. 441-452, 1995.
[8] Conference of the Rectors of D16 Universities, “Dati Universitari 1992/93” (in D16), Conference
of the Rectors of D16 Universities, Roma, I, 1996.
[9] CPLEX Optimization, Inc. CPLEX, CPLEX Optimization, Inc., Incline Village, NV, 1994.
16
[10] J. Doyle and R. Green, “Efficiency and cross-efficiency in DEA: derivation, meaning and
uses”, Journal of the Operational Research Society, vol. 45, No. 5, pp. 567-578, 1994.
[11] M. K. Epstein and J. C. Henderson, “Data Envelopment Analysis for managerial control and
diagnosis”, Decision Sciences, vol. 20, pp. 90-119, 1989.
[12] P. N. Finlay and G. Gregory, “A management support system for directing and monitoring
the activities of university academic staff”, Journal Operational Research Society, vol. 45, No.
6, pp. 641-650, 1994.
[13] H. Fried, K. Lovell, and S. Schmidt, “The measurement of productive efficiency: techniques
and applications”, Oxford University Press, New York, 1993.
[14] GAMS Development Corp. CPLEX, GAMS Development Corp., Washington, D.C., 1994.
[15] M. Norman and B. Stoker, Data Envelopment Analysis, Wiley, Chichester, UK, 1991.
[16] N. C. Petersen, “Data envelopment analysis on a relaxed set of assumptions”, Management
Science, vol. 36, No. 3, pp. 305-314, 1990.
[17] R. Pesenti, and W. Ukovich “ Data Envelopment Analysis: a possible way to evaluate the
academic activities — an overview”, Internal Report, DEEI, UTS, 1996.
[18] Z. Sinuany-Stern, A. Mehrez, and A. Barboy, “Academic departments efficiency via DEA”,
Computers and Operations Research, vol. 5, pp. 543-554, 1994.
[19] E. Thanassoulis, R. G. Dyson, and M. J. Foster, “Relative efficiency assessment using data
envelopment analysis: An application to data on rates departments”, J. Opl. Res. Soc., vol.
38, No. 5, pp. 397-411, 1987.
[20] E. Thanassoulis, and R. G. Dyson, “Estimating preferred target input-output levels using
Data Envelopment Analysis”, European Journal of Operational Research, vol. 56, pp. 80-97,
1992.
[21] H. Tulkens and P. H. Eeckaut, “Non-parametric efficiency, progress and regress measures for
panel data: Methodological aspects”, European Journal of Operational Research, vol. 80, pp.
474-499, 1995.
[22] University of Warwick, “DEA models”, http://www.csv.warwick.ac.uk/~bsrlu/, 1996.
17