On Bias, Variance, 0/1-Loss, and
the Curse-of-Dimensionality-2nd
Weiqiang Dong
1
Function Estimate
• Input: 𝒙
• Output: 𝑦 = 𝑓 𝒙 + 𝜀
where 𝑓(𝒙) (“target function”) is a single valued deterministic function
of 𝒙 and 𝜀 is a random variable, 𝐸(𝜀 | 𝒙) = 0.
• The goal is to obtain an estimate
using a training data set T
2
Estimation Error
• The goal is to obtain an estimate
using a training data set T
• Mean Square Error:
𝐸𝑇[𝑦 − 𝑓(𝒙| 𝑇)]2
= [𝑓(𝒙) − 𝐸𝑇 𝑓(𝒙| 𝑇)]2 +𝐸𝑇 [𝑓(𝒙| 𝑇) − 𝐸𝑇 𝑓(𝒙| 𝑇)]2 + 𝐸𝜀 [𝜀| 𝒙]2
1. Square of bias
2. Variance
3. Irreducible prediction error
3
• 𝐸𝑇[𝑦 − 𝑓(𝒙| 𝑇)]2
= [𝑓(𝒙) − 𝐸𝑇 𝑓(𝒙| 𝑇)]2 + 𝐸𝑇 [𝑓(𝒙| 𝑇) − 𝐸𝑇 𝑓(𝒙| 𝑇)]2 + 𝐸𝜀 [𝜀| 𝒙]2
1. Square of bias
2. Variance
3. Irreducible prediction error
1. Square of bias: The extent to which the average prediction over all
data sets differs from the desired regression function.
2. Variance: The extent to which the solutions for individual data sets
vary around their average (sensitivity to the particular choice of
data set).
4
• 𝐸𝑇[𝑦 − 𝑓(𝒙| 𝑇)]2
= [𝑓(𝒙) − 𝐸𝑇 𝑓(𝒙| 𝑇)]2 + 𝐸𝑇 [𝑓(𝒙| 𝑇) − 𝐸𝑇 𝑓(𝒙| 𝑇)]2 + 𝐸𝜀 [𝜀| 𝒙]2
1. Square of bias
2. Variance
3. Irreducible prediction error
Red Dot: True model
Blue Dot: Individual estimation
It is desirable to have both low biassquared and low variance since
both contribute to the squared
estimation error in equal measure.
5
Classification
• Input: 𝒙 = {𝑥1, … , 𝑥𝑛}
• Output: 𝑦 ∈ {0, 1}
• Prediction 𝑦 ∈ {0, 1}
• The goal is to choose 𝑦 𝒙 to minimize inaccuracy as characterized by
the misclassification “risk”
6
• The goal is to choose 𝑦 𝒙 to minimize inaccuracy as characterized by
the misclassification “risk”
• The misclassification risk (2.2) is minimized by the (“Bayes”) rule
which achieves the lowest possible risk
7
• Let 𝑙0 = 𝑙1 = 1, the misclassification risk is minimized by the (“Bayes”)
rule
1
𝑦𝐵 𝒙 = 1 𝑓(𝒙) ≥
2
Let 𝑔(𝑥) = 𝑓(𝑥) − 1/2,
𝑦𝐵 𝒙 = 1 𝑔(𝒙) ≥ 0
𝑔(𝑥) can be considered as the decision boundary.
8
• Let 𝑙0 = 𝑙1 = 1, the misclassification risk is minimized by the (“Bayes”)
rule
1
𝑦𝐵 𝒙 = 1 𝑓(𝒙) ≥
2
Let 𝑔(𝑥) = 𝑓(𝑥) − 1/2,
𝑦𝐵 𝒙 = 1 𝑔(𝒙) ≥ 0
𝑦 𝒙 = 1 𝑔(𝒙|𝑻) ≥ 0
𝑔(𝑥) can be considered as the decision boundary.
𝑔 𝒙|𝑻 is the estimation of the decision boundary.
9
Classification
• Input: 𝒙
• Output: 𝑦 ∈ {0, 1}
• Decision boundary: g(𝑥)
y 𝒙 = 1 𝑔(𝒙) ≥ 0
10
Classification
• Input: 𝒙
• Output: 𝑦 ∈ {0, 1}
• Decision boundary: g(𝑥)
• Prediction 𝑦 ∈ {0, 1}
• Estimate of decision boundary: 𝑔 𝒙 𝑇)
y 𝒙 = 1 𝑔(𝒙) ≥ 0
𝑦 𝒙 = 1 𝑔 𝒙 𝑻) ≥ 0
The training data set T is used to learn a classification rule
y(𝑥 | 𝑇 ) for (future) prediction. The usual paradigm for
accomplishing this is to use the training data T to form an
approximation (estimate) 𝑔(𝑥 | 𝑇 ) to 𝑔(𝑥)
Regular function estimation technology can be applied to
obtain the estimate 𝑔 𝑥 𝑇)
11
Estimation Error
𝐸𝑇[𝑦 − 𝑓(𝒙| 𝑇)]2 = [𝑓(𝒙) − 𝐸𝑇 𝑓(𝒙| 𝑇)]2 + 𝐸𝑇 [𝑓(𝒙| 𝑇) − 𝐸𝑇 𝑓(𝒙| 𝑇)]2 + 𝐸𝜀 [𝜀| 𝒙]2
1. Square of bias
2. Variance
3. Irreducible prediction error
• It is desirable to have both low bias-squared and low variance since both
contribute to the squared estimation error in equal measure.
Classification Error
𝑦𝐵 𝒙 = 1 𝑔(𝒙) ≥ 0
𝑦 𝒙 = 1 𝑔(𝒙|𝑻) ≥ 0
• The estimate 𝐸 𝑔 may be off from 𝑔 by a huge margin. It does not matter as long
as we take care of the fact that 1 𝐸 𝑔 𝒙 𝑇) ≥ 0 =1 𝑔(𝒙) ≥ 0 and cut down our
variance.
12
Output: y 𝒙 = 1 𝑔(𝒙) ≥ 0
Prediction: 𝑦(𝒙) = 1 𝑔 𝒙 𝑇) ≥ 0
Red Line: 𝑔(𝑥)
Blue line: 𝑔 𝒙 𝑇)
13
Output: y 𝒙 = 1 𝑔(𝒙) ≥ 0
Prediction: 𝑦(𝒙) = 1 𝑔 𝒙 𝑇) ≥ 0
Red Line: 𝑔(𝑥)
Blue line: 𝑔 𝒙 𝑇)
14
Output: y 𝒙 = 1 𝑔(𝒙) ≥ 0
Prediction: 𝑦(𝒙) = 1 𝑔 𝒙 𝑇) ≥ 0
Red Line: 𝑔(𝑥)
Blue line: 𝑔 𝒙 𝑇)
The estimate 𝐸 𝑔 may be off from 𝑔 by
a huge margin. It does not matter as
long as we take care of the fact that
1 𝐸 𝑔 𝒙 𝑇) ≥ 0 =1 𝑔(𝒙) ≥ 0 and
cut down our variance.
15
Estimation Error
𝐸𝑇[𝑦 − 𝑓(𝒙| 𝑇)]2 = [𝑓(𝒙) − 𝐸𝑇 𝑓(𝒙| 𝑇)]2 + 𝐸𝑇 [𝑓(𝒙| 𝑇) − 𝐸𝑇 𝑓(𝒙| 𝑇)]2 + 𝐸𝜀 [𝜀| 𝒙]2
1. Square of bias
2. Variance
3. Irreducible prediction error
• It is desirable to have both low bias-squared and low variance since both
contribute to the squared estimation error in equal measure.
Classification Error
𝑦𝐵 𝒙 = 1 𝑔(𝒙) ≥ 0
𝑦 𝒙 = 1 𝑔(𝒙|𝑻) ≥ 0
• The estimate 𝐸 𝑔 may be off from 𝑔 by a huge margin. It does not matter as long
as we take care of the fact that 1 𝐸 𝑔 𝒙 𝑇) ≥ 0 =1 𝑔(𝒙) ≥ 0 and cut down our
variance.
The aggregation approach often decreases the variance and slightly increases the
bias.
16
The aggregation approach often improve classification accuracy since it reduces the
variance.
single decision tree vs bagging of tree
𝑓(𝑥)
17
The aggregation approach often improve classification accuracy since it reduces the
variance.
single decision tree vs bagging of tree
𝑓(𝑥)
𝑓 𝑥 + 𝑛𝑜𝑖𝑠𝑒
18
The aggregation approach often improve classification accuracy since it reduces the
variance.
single decision tree vs bagging of tree
𝑓(𝑥)
𝑓 𝑥 + 𝑛𝑜𝑖𝑠𝑒
𝑓 𝒙 𝑇)
19
The aggregation approach often improve classification accuracy since it reduces the
variance.
single decision tree vs bagging of tree
𝑓(𝑥)
𝑓 𝑥 + 𝑛𝑜𝑖𝑠𝑒
𝑓 𝒙 𝑇)
𝐸𝑓
20
The aggregation approach often improve classification accuracy since it reduces the
variance.
single decision tree vs bagging of tree
𝑓(𝑥)
𝑓 𝑥 + 𝑛𝑜𝑖𝑠𝑒
𝑓 𝒙 𝑇)
𝐸𝑓
21
𝑓(𝑥)
𝑓 𝑥 + 𝑛𝑜𝑖𝑠𝑒
𝑓 𝒙 𝑇)
𝐸𝑓
Tree: 0.0255 (error) = 0.0003 (bias^2) + 0.0152 (var) + 0.0098 (noise)
Bagging(Tree): 0.0196 (error) = 0.0004 (bias^2) + 0.0092 (var) + 0.0098 (noise) 22
Estimation Error
𝐸𝑇[𝑦 − 𝑓(𝒙| 𝑇)]2 = [𝑓(𝒙) − 𝐸𝑇 𝑓(𝒙| 𝑇)]2 + 𝐸𝑇 [𝑓(𝒙| 𝑇) − 𝐸𝑇 𝑓(𝒙| 𝑇)]2 + 𝐸𝜀 [𝜀| 𝒙]2
1. Square of bias
2. Variance
3. Irreducible prediction error
• It is desirable to have both low bias-squared and low variance since both
contribute to the squared estimation error in equal measure.
Classification Error
𝑦𝐵 𝒙 = 1 𝑔(𝒙) ≥ 0
𝑦 𝒙 = 1 𝑔(𝒙|𝑻) ≥ 0
• The estimate 𝐸 𝑔 may be off from 𝑔 by a huge margin. It does not matter as long
as we take care of the fact that 1 𝐸 𝑔 𝒙 𝑇) ≥ 0 =1 𝑔(𝒙) ≥ 0 and cut down our
variance.
The aggregation approach often decreases the variance and slightly increases the
bias.
23
• Much research in classification has been devoted to achieving higher
accuracy probability estimates under the presumption that this will
generally lead to more accurate predictions. This need not always be
the case.
24