Chapter 15
System Errors Revisited
Ali Erol
10/19/2005
1
System Errors Revisited
• Quantify the accuracy of FAR and FRR
estimates.
• Confidence Intervals, a well known technique
used in statistical analysis.
• See references [22],[23].
• The first three author’s algorithm [23]
experimentally demonstrated to provide better
Confidence Intervals estimates.
2
FAR/FRR
• Definition:
FRR(x)=Prob(smx/H0)=F(x)
FAR(y)=Prob(sn>y/Ha)=1-Prob(sn  y/Ha)=1-G(y)
• We need
– F(x)=Dist(x) : Genuine (Matching) score DF
– G(y)= Dist(y): Imposter (Non-matching) score DF
3
FAR/FRR
• Instead we have
– Set of genuine scores X={X1, X2, …., XM}
– Set of imposter scores Y={Y1,Y2, …., YN}
• We estimate
^
1
FRR ( x ) 
(# X i  x )
M
^
1
FAR ( y )  (# Yi  y )
N
4
Problem
• What is the accuracy of these error rates?
– The number of biometric samples
– The quality of the samples
• Data collection procedure (e.g. 10 consecutive samples)
• Subjects involved, the acquisition device etc.
5
An Estimation Problem
Given
x: A random variable (F(x) denotes Dist(x))
X={X1, X2, …., XM}: Sample set
Estimate
=E(x)
Solution
1
ˆ
X
M
Error
M
X
i 1
i
(Unbiased estimator*)
r  ˆ  
6
Biased/Unbiased Estimators
• For an unbiased estimator we have
E (ˆ /  )  
• Example: Gaussian Model: Estimate mean 1
and variance 2 using maximum likelihood
criterion i.e. maximize Prob(X/ ,)
M
1
ˆ1   X i
M i 1
1 M
ˆ
 2   ( X i  ˆ1 ) 2
M i 1
M
1
ˆ2 
( X i  ˆ1 ) 2

M  1 i 1
(Unbiased estimator)
(Biased estimator)
(Unbiased estimator)
M 1
E (ˆ2 / θ) 
2
M
7
Confidence Interval
• Assume F(x) is given then Dist(r) can be
calculated
– r is function of ˆ, which is a function of x
• Calculate (1-) 100% certainty (Next Slide)
r[1(,X), 2(,X)]
• Which leads to (1-)100% confidence
interval for  given by
  [ˆ   2 ( , X ), ˆ  1 ( , X )]
8
Confidence Interval
• Example
– Discard /2 on lower and higher ends
– Find the r values corresponding to the interval
boundary (called quantile)
Dist(r)
r
Prob(q(/2) r q(1-/2))=1-
9
Confidence Interval
• Interpretation:
– Generate sample sets X from F(x)
– Calculate confidence intervals for each X
– (1-)100% of these intervals contain .
10
Parametric Method
1
ˆ
X
M
M
X
i 1
i
• Xi identically distributed
• Assume Xi are independent (not true in general)
• Then Dist (ˆ) can be taken to be normal distribution
using central limit theorem (large M).
• Result:
E (ˆ)  X
stddev( X)
X

z
(

)
Var
(
X
)
Var(ˆ) 
M
M
• E.g. For 95% confidence z=1.96
• Smaller interval with increasing M and 
11
Non-Parametric Method
• Assume F(x) is available.
Sample Set
X
f(x)
Density of
X*
Additional
Sample Sets
X*i i  [1...B ]
Random
Variable
X*
12
Non-Parametric Method
• FACT: For large B we have
E( X )  E( x)
• Define error to be
r  X*  X
*
• Calculate Dist(r)
• Solution:
1
Dist ( r )  (# ( X i*  X )  r )
B
13
Non-Parametric Method
• Interval calculation: Sorting and counting
Dist(r)
r
/2
X i* i  [1...B] /2
X 1*  X 2*  .......  X B*
q* ( / 2)  X k*1 and q* (1   / 2)  X k*2
k1  ( / 2) B and k2  1  ( / 2) B
14
Bootstrap Method
• F(x) is not available; all we have is X
• How do we generate X i* ?
• Solution (i.e. Bootstrap method): Sampling
with replacement from X.
• Put the samples in a bag, draw, record and put
it back.
• Draw M samples from X B times. Some
samples Xi may not be in each set.
15
Bootstrap Method (Imperfections)
• Xi are not independent.
– In SR the dependence between samples is not
replicated.
• Effect of dependence for independent samples
– Variance of X *is smaller
X 1 , X 2 zero mean i.d
  when independen t
E ( X  X )   4 E X  when X  X
E ( X 1  X 2 ) 2   2 E X 1
2
2
1
2
2
1
1
2
– Leads to smaller CIs
/2
X i* i  [1...B] /2
16
Subset Bootstrap
• Potential sources of dependency
– All samples from the same person (e.g. multiple
fingers)
– All samples from same biometric (e.g. finger)
• Partition X into independent subsets
• Apply SR on subsets.
17
Subset Bootstrap (An example)
• Fingerprint database
–
–
–
–
P persons
c fingers per person  D=cP Fingers
d samples per finger
DB Size= cPd
• Matching pairs
– d(d-1) per finger
– cd(d-1) per person
– cPd(d-1)=Dd(d-1) total
• Using a symmetric and asymmetric matcher does not
make any difference [23].
18
Subset Bootstrap (An Example)
X1 X2
X1: P=10 c=2, D=20, d=8  M=1120
X2: P=50 c=2, D=100, d=8  M=5600
Finger based partition: Set subsets to be the
samples from the same finger (i.e. D subsets of
d(d-1) matching scores)
• Person based partition: Set subsets to be the
samples from the same person (i.e. P subsets
of cd(d-1) matching scores)
•
•
•
•
19
Subset Bootstrap (An Example)
• We expect
– CI1 (light gray) to be larger than CI2 (dark gray)
• Because X1 has smaller number of samples
– CI2 (dark gray) to be contained in CI1 (light gray)
• Because X1 X2
• The intervals are larger for person based partitioning
– There is dependency between fingers of the same person
20
CIs for FAR/FRR
• Calculate CIs for each
threshold T=t0 and given
an 
1
FRR (t0 ) 
(# X i  t0 )
M
1
FAR (t0 )  (# Yi  t0 )
N
21
CI for FRR
• Given genuine score set X
– Generate X i  [1...B]
– Calculate FRR *i (t0 )
– Sort and count
*
i
FRR (t0 ; )  [ FRR*k1 , FRR*k2 ]
k1  ( / 2) B and k2  1  ( / 2) B
22
CI for FAR
• Given imposter score set Y
– Generate Y i  [1...B]
– Calculate FAR *i (t0 )
– Sort and count
*
i
FAR (t0 ; )  [ FAR*k1 , FAR*k2 ]
k1  ( / 2) B and k2  1  ( / 2) B
23
Subset Bootstrap for FAR
• Imposter scores Y are not independent
• We are using multiple impressions of the same
finger.
• Let Ixk: kth finger impression from subject x
then sim(Ia1,Ib1), sim(Ia1,Ib2), sim(Ia2,Ib3) are not
statistically independent
• Use a finger only once; for D fingers we have
only D/2 such pairs
• There is actually dependency between X and Y
24
Subset Bootstrap for FAR
• Fingerprint database
–
–
–
–
P persons
c fingers per person  D=cP Fingers
d samples per finger
DB Size= cPd
• Non-matching pairs
– N=d2D(D-1)=P[(dc)2(P-1)+d2c(c-1)]
– d2(D-1) per finger
– (dc)2(P-1)+d2c(c-1) per person
25
Subset Bootstrap for FAR
DB Partition
I1
….
Ii
….
IN
x
Y1=IixI1
Ii
YN-1=IixIN
• Finger (N=D): Take Ii (d elements), match it against Iki (d2 pairs)
then we have d2(D-1) pairs. Repeat it with all Ii to construct
subsets Yk
• Person (N=P): Take Ii (cd elements), match it against Iki ((dc)2
pairs) then we have (dc)2(P-1) pairs. Inside Ii we have d2c(c-1)
pairs. Repeat it with all Ii to construct subsets Yk
26
• Not completely independent: We use Ii many times.
Subset Bootstrap for FRR
•
Person subset is a better estimate
27
How good are the CIs?
• There exists a true confidence interval (At the
beginning we assumed F(x) is known)
• The CI we calculate is just one estimate.
• How accurate is that estimate?
28
How good are the CIs?
•
•
We estimate E(x)
Ideal Test: Assume F(x) is available
–
–
–
Generate X k k  [1...K ]
Calculate X k
Assume E( x)  X and test if X k  CI
29
How good are the CIs?
• Practical Test (for comparison)
1. Randomly split X into two subsets Xa and Xb
2. Calculate X b and CIa
3. Test X b  CIa
4. Repeat 1-3 many times and count the number of hits i.e.
probability of X bfalling into the CIa
• Hit rate is not equal to the confidence. Assume X b , X a
have normal distribution.
• The higher the hit rate is the better the estimates are.
30
How good are the CIs?
• =0.1
• Person based partitioning provide more
accurate confidence intervals
• 73.10% is very close to the expected value
31