The use of fuzzy relations in the assessment of
information resources producers' performance
,
1 2
Marek Gagolewski
1
3
Systems Research Institute, Polish Academy of Sciences,
ul. Newelska 6, 01-447 Warsaw, Poland,
2
Jan Lasek
[email protected]
Faculty of Mathematics and Information Science, Warsaw University of Technology,
ul. Koszykowa 75, 00-662 Warsaw, Poland
3
Interdisciplinary PhD Studies Program,
Systems Research Institute, Polish Academy of Sciences
[email protected]
IEEE Intelligent Systems
Warsaw, Poland 2014
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6, 101-447
/ 21 W
Introduction and motivation
In today's world with massive amounts of data we suer from so-called
information overload.
There is a need for methods for selection of valuable
producers.
The goals of our study are twofold:
to overcome some of the drawbacks of standard approaches for
evaluation of producers' outcomes and
to provide tools for selecting interesting producers.
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6, 201-447
/ 21 W
Producer Assessment Problem (PAP)
Formal denition of the problem under our consideration [Gagolewski and
Grzegorzewski 2011].
Producer Assessment Problem
P = {p1 , . . . , pk } be a nite set consisting of k producers. The i -th
producer outputs ni products. Additionally, each product is given some kind
Let
of quantitative rating, e.g. concerning its overall quality.
The state of
pi
may be described by a sequence
(i)
x
(i)
∈ I1,2,... =
= x1 , . . . , xn(i)
i
[
In
n­1
with elements in
I,
e.g.
I = [0, ∞).
Most importantly, we should note that
the numbers of products may vary from producer to producer.
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6, 301-447
/ 21 W
Standard approach toward PAP (1)
The standard approach toward PAP is the use of so-called impact functions
[Gagolewski and Grzegorzewski 2011, Gagolewski 2013, Quesada 2010,
Woeginger 2008].
Denition (Impact function)
Impact function F
: I1,2,... → I
is a function with the properties that it is
non-decreasing in each variable,
arity-monotonic (additional product unit(s) does not reduce the overall
producer evaluation),
symmetric.
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6, 401-447
/ 21 W
Standard approach toward PAP (2)
With the class of impact functions the following relation is connected.
a relation
E ⊆ I1,2,... ×
E y iff n ¬ m
and
x(i) ¬ y(i)
for all
i ¬ n,
2
x
x
y
6
I
∈ In , y ∈ Im
, ,... be dened as
1 2
4
Let for x
8
Denition (Producers' dominance relation)
x(i)
denotes the i -th greatest coordinate of
0
where
vector x.
1
2
3
4
5
6
7
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6, 501-447
/ 21 W
Standard approach toward PAP (2)
With the class of impact functions the following relation is connected.
a relation
E ⊆ I1,2,... ×
E y iff n ¬ m
and
x(i) ¬ y(i)
for all
i ¬ n,
2
x
x
y
6
I
∈ In , y ∈ Im
, ,... be dened as
1 2
4
Let for x
8
Denition (Producers' dominance relation)
x(i)
denotes the i -th greatest coordinate of
0
where
vector x.
1
2
3
4
5
6
7
In fact, the following theorem applies [Gagolewski and Grzegorzewski 2011]:
Theorem
Let
F
: I1,2,... → I
be an aggregation operator. Then
F is symmetric,
nondecreasing with respect to each variable and arity-monotonic if and only
, ∈ I1,2,...
if for any x y
if x
E y,
then
F(x)
¬ F(y).
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6, 501-447
/ 21 W
Issues with the standard approach toward PAP
There are several issues that come with the standard approaches toward
PAP:
Embedding of producers' output into, e.g.,
order induced by
¬
I = [0, ∞),
where linear
is present, gives the comparison between producers
out of control.
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6, 601-447
/ 21 W
Issues with the standard approach toward PAP
There are several issues that come with the standard approaches toward
PAP:
Embedding of producers' output into, e.g.,
order induced by
¬
I = [0, ∞),
where linear
is present, gives the comparison between producers
out of control.
If one wants to assign equal scores to incomparable cases w.r.t.
then the impact function needs to be trivial, i.e,
c ∈I
∀x F(x) = c
E
for some
[Gagolewski 2013, Theorem 3].
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6, 601-447
/ 21 W
Issues with the standard approach toward PAP
There are several issues that come with the standard approaches toward
PAP:
Embedding of producers' output into, e.g.,
order induced by
¬
I = [0, ∞),
where linear
is present, gives the comparison between producers
out of control.
If one wants to assign equal scores to incomparable cases w.r.t.
then the impact function needs to be trivial, i.e,
c ∈I
∀x F(x) = c
E
for some
[Gagolewski 2013, Theorem 3].
One can arrive at any desired ranking of producers by an appropriate
construction
of an impact function [Gagolewski 2013, Theorem 4].
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6, 601-447
/ 21 W
Fuzzy approach toward PAP
First, we recall the denition of a fuzzy relation.
Denition (Fuzzy relation)
A is a pair (R, µ), where µ is the membership
R , µ : A × A → [0, 1], measuring the degree to which R holds.
A fuzzy relation on the set
function of
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6, 701-447
/ 21 W
Fuzzy approach toward PAP
First, we recall the denition of a fuzzy relation.
Denition (Fuzzy relation)
A is a pair (R, µ), where µ is the membership
R , µ : A × A → [0, 1], measuring the degree to which R holds.
A fuzzy relation on the set
function of
We say that a relation
(fuzzy) reexive if
R
is
µ(a, a) = 1
for all
a ∈ A,
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6, 701-447
/ 21 W
Fuzzy approach toward PAP
First, we recall the denition of a fuzzy relation.
Denition (Fuzzy relation)
A is a pair (R, µ), where µ is the membership
R , µ : A × A → [0, 1], measuring the degree to which R holds.
A fuzzy relation on the set
function of
We say that a relation
(fuzzy) reexive if
R
is
µ(a, a) = 1
for all
a ∈ A,
additive reciprocal (or probabilistic) if
µ(a, b) + µ(b, a) = 1,
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6, 701-447
/ 21 W
Fuzzy approach toward PAP
First, we recall the denition of a fuzzy relation.
Denition (Fuzzy relation)
A is a pair (R, µ), where µ is the membership
R , µ : A × A → [0, 1], measuring the degree to which R holds.
A fuzzy relation on the set
function of
We say that a relation
(fuzzy) reexive if
R
is
µ(a, a) = 1
for all
a ∈ A,
additive reciprocal (or probabilistic) if
is fuzzy
T -transitive
if for any
µ(a, b) + µ(b, a) = 1,
a, b ∈ A
µ(a, b) ­ sup T (µ(a, c), µ(c, b)),
c∈A
where
T
is a given t-norm.
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6, 701-447
/ 21 W
Fuzzy approach toward PAP
First, we recall the denition of a fuzzy relation.
Denition (Fuzzy relation)
A is a pair (R, µ), where µ is the membership
R , µ : A × A → [0, 1], measuring the degree to which R holds.
A fuzzy relation on the set
function of
We say that a relation
(fuzzy) reexive if
R
is
µ(a, a) = 1
for all
a ∈ A,
additive reciprocal (or probabilistic) if
is fuzzy
T -transitive
if for any
µ(a, b) + µ(b, a) = 1,
a, b ∈ A
µ(a, b) ­ sup T (µ(a, c), µ(c, b)),
c∈A
where
T
is a given t-norm.
These notions have their natural counterparts in crisp setting.
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6, 701-447
/ 21 W
10
Exemplary class of fuzzy preference relation for PAP (1)
0
2
4
6
8
x
y
1
2
3
4
5
6
7
8
9
10
11
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6, 801-447
/ 21 W
10
Exemplary class of fuzzy preference relation for PAP (1)
0
2
4
6
8
x
y
1
2
3
4
5
6
7
8
9
10
11
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6, 901-447
/ 21 W
10
Exemplary class of fuzzy preference relation for PAP (1)
x
y
4
6
8
πxy
0
2
πyx
1
2
3
4
5
6
7
8
9
10
11
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1001-447
/ 21 W
Exemplary class of fuzzy preference relation for PAP (2)
S := space of innite nonincreasing sequences with elements in I.
Let e
· : I1,2,... → S be an operator such that for x ∈ In we have
e
x = (x(n) , x(n−1) , . . . , x(1) , 0, 0, . . . ).
Denition (Fuzzy producers' dominance relation)
Let x, y
∈ S,
and w
dominance relation
= (w1 , w2 , . . . ), wi > 0
for all
is a fuzzy preference relation
J
i.
The fuzzy producers
with the membership
function given by:
µ(x, y) =
where
πxy =
P
i


πyx
πxy +πyx
0.5
if
πxy + πyx > 0,
otherwise,
wi · max{xi − yi , 0}.
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1101-447
/ 21 W
Properties
The proposed relation has the following properties:
it is additive reciprocal:
µ(x, y) + µ(y, x) = 1,
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1201-447
/ 21 W
Properties
The proposed relation has the following properties:
it is additive reciprocal:
µ(x, y) + µ(y, x) = 1,
it is (fuzzy) transitive under Šukasiewicz T-norm,
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1201-447
/ 21 W
Properties
The proposed relation has the following properties:
it is additive reciprocal:
µ(x, y) + µ(y, x) = 1,
it is (fuzzy) transitive under Šukasiewicz T-norm,
however, it is not fuzzy reexive:
µ(x, x) = 0.5,
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1201-447
/ 21 W
Properties
The proposed relation has the following properties:
it is additive reciprocal:
µ(x, y) + µ(y, x) = 1,
it is (fuzzy) transitive under Šukasiewicz T-norm,
however, it is not fuzzy reexive:
when x
6= y
and x
Ey
then
µ(x, x) = 0.5,
µ(x, y) = 1,
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1201-447
/ 21 W
Properties
The proposed relation has the following properties:
it is additive reciprocal:
µ(x, y) + µ(y, x) = 1,
it is (fuzzy) transitive under Šukasiewicz T-norm,
however, it is not fuzzy reexive:
when x
6= y
and x
Ey
then
µ(x, x) = 0.5,
µ(x, y) = 1,
Hence, the relation is fuzzy preference relation in the sense studied by, e.g.
Tanino [1988].
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1201-447
/ 21 W
Aggregation of pairwise comparisons to a ranking
Producers' scores may be derived by e.g. the net ow method [Bouyssou
1992, Fodor and Roubens 1994]. This method assigns scores according to
the formula
Snet (xi ) =
X
µ(xj , xi ) − µ(xi , xj ) = “inflow ” − “outflow ”
xj ∈X
Producers are ranked with respect to their scores. This is quite analogous
to the classical approach in which the impact functions are used.
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1301-447
/ 21 W
Quality of rankings
We also would like to suggest an evaluation (quality) measure
ranking
r.
Q
for
We require that the measure has at least the following
properties:
1
Q(r , J) ∈ [0, 1],
where we assume that 0 and 1 are the lowest and the
highest possible quality value, respectively;
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1401-447
/ 21 W
Quality of rankings
We also would like to suggest an evaluation (quality) measure
ranking
r.
Q
for
We require that the measure has at least the following
properties:
1
Q(r , J) ∈ [0, 1],
where we assume that 0 and 1 are the lowest and the
highest possible quality value, respectively;
2
The following ranking is of the highest quality (1)
σ(1)
x
1
1
1
I xσ(2) I · · · I xσ(k)
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1401-447
/ 21 W
Quality of rankings
We also would like to suggest an evaluation (quality) measure
ranking
r.
Q
for
We require that the measure has at least the following
properties:
1
Q(r , J) ∈ [0, 1],
where we assume that 0 and 1 are the lowest and the
highest possible quality value, respectively;
2
The following ranking is of the highest quality (1)
σ(1)
x
3
1
1
1
I xσ(2) I · · · I xσ(k)
The lowest score (0) is assigned to ranking
σ(1)
x
0
0
0
I xσ(2) I · · · I xσ(k)
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1401-447
/ 21 W
Quality of rankings
We also would like to suggest an evaluation (quality) measure
ranking
r.
Q
for
We require that the measure has at least the following
properties:
1
Q(r , J) ∈ [0, 1],
where we assume that 0 and 1 are the lowest and the
highest possible quality value, respectively;
2
The following ranking is of the highest quality (1)
σ(1)
x
3
1
1
1
I xσ(2) I · · · I xσ(k)
The lowest score (0) is assigned to ranking
σ(1)
x
0
0
0
I xσ(2) I · · · I xσ(k)
The following function can constitute an exemplary quality measure:
P
Q(r , J) =
i,j:
r (xi )>r (xj )
µ(xi , xj ) +
P
i<j:
r (xi )=r (xj )
n
1
− 2 µ(xi , xj ) − 21 .
2
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1401-447
/ 21 W
Application - Ranking of users at StackOverow (1)
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1501-447
/ 21 W
Application - Ranking of users at StackOverow (2)
We applied chosen methods to rank 100 most active users on
StackOverow (with the biggest number of answers).
We confronted our results with standard approaches using
StackOverow reputation index iR ,
average quality of an answer x̄,
maximal quality answer
x(n) ,
sum of quality over answers
number of answers
Hirsch's
h-index iH
Σ(x),
n,
w -index iW ,
optimization (SO ).
and Woeninger
rankings found by stochastic
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1601-447
/ 21 W
Application - Ranking of users at StackOverow (2)
We applied chosen methods to rank 100 most active users on
StackOverow (with the biggest number of answers).
We confronted our results with standard approaches using
StackOverow reputation index iR ,
average quality of an answer x̄,
maximal quality answer
x(n) ,
sum of quality over answers
number of answers
Hirsch's
h-index iH
Σ(x),
n,
w -index iW ,
optimization (SO ).
and Woeninger
rankings found by stochastic
Table: Quality measures of rankings.
iR
x̄
x(n)
Σ(x)
n
iH
iW
0.895
0.748
0.749
0.88
0.726
0.831
0.819
NF
0.874
SO
0.914
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1601-447
/ 21 W
Future work - learning relations from data: questionnaire
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1701-447
/ 21 W
Summary and future work
In our study we employed tools from fuzzy logic and fuzzy set theory
to Producer Assessment Problem. We argue that it is a more gentle
approach for evaluation of producers.
The fuzzy pairwise comparison relation allows us to naturally extend
its counterpart in the crisp setting in which many pairs of producers
are incomparable.
The proposed relation founds a basis of the ranking and is the most
important ingredient of the model. Such a relation might be
constructed not only by an explicit formula, but by statistical or
machine learning models.
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1801-447
/ 21 W
Thank you!
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
1901-447
/ 21 W
Bibliography I
Bouyssou, D.: Ranking methods based on valued preference relations:
A characterization of the net ow method. European Journal of
Operational Research, 60:6067, 1992.
Cena, A., Gagolewski, M.: OM3: Ordered maxitive, minitive, and
modular aggregation operators Axiomatic and probabilistic properties
in an arity-monotonic setting. Fuzzy Sets and Systems, 4, 2014.
Egghe, L.: The Hirsch index and related impact measures. Annual
Review of Information Science and Technology, 44:65114, 2010.
Fodor, J., Roubens, M.: Fuzzy Preference Modelling and Multicriteria
Decision Support. Springer, 1994.
Gagolewski, M. and Grzegorzewski, P.: Possibilistic analysis of
arity-monotonic aggregation operators and its relation to bibliometric
impact assessment of individuals. International Journal of Approximate
Reasoning, 52(9):13121324, 2011.
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
2001-447
/ 21 W
Bibliography II
Gagolewski M.: Scientic impact assessment cannot be fair. Journal of
Informetrics 7(4):792-802, 2013.
Hirsch, J. E.: An index to quantify individual's scientic research
output. Proceedings of the National Academy of Sciences,
102(46):16569 16572, 2005.
Quesada, A.: More axiomatics for the Hirsch index. Scientometrics,
82:413418, 2010.
Tanino, T.: Fuzzy Preference Relations in Group Decision Making,
5471. Springer Berlin Heidelberg, 1988.
Woeginger, G. J.: A symmetry axiom for scientic impact indices.
Journal of Informetrics, 2:298303, 2008.
M. Gagolewski and J. Lasek (SRI PAS) (SystemsFuzzy
Research
methods
Institute,
in PAP
Polish Academy
Intelligent
of Sciences,
Systems
ul. Newelska
2014
6,
2101-447
/ 21 W