Fundamentals of the KRLS Algorithm
2016/12/2
introduction
• Kernel machines are a relatively new class of learning algorithms(2003~2004)
utilizing Mercer kernels in order to produce non-linear versions of conventional
linear supervised and unsupervised learning algorithms.
kernal methods
• Low dimensions may become much easier if the data is mapped to a
high-dimensional space.
online sparsification
• To avoid adding the training sample 𝑥𝑡 to the dictionary, we need to
find coefficients 𝑎𝑡 = (𝑎1 , … , 𝑎𝑚𝑡−1 )𝑇 satisfying the approximate
linear dependence (ALD) condition：
where 𝜈 is the sparsity level parameter.
• If 𝛿𝑡 ≤ 𝜈, 𝜙(𝑥𝑡 )can be approximated within a squared error 𝜈 by
some linear combination of dictionary instances.
• And using 𝑘 𝑥𝑖 , 𝑥𝑗 =〈𝜙(𝑥𝑖 ),𝜙(𝑥𝑗 )〉, ALD can write：
where
dictionary samples,
is the kernel matrix calculated with the
and
.
,
,for 𝑖, 𝑗 = 1, … , 𝑚𝑡−1
• The solution of ALD is given by
for which we have
• If otherwise 𝛿𝑡 > 𝜈 , the current dictionary must be expanded by
adding 𝑥𝑡 .
Thereby,
and
.
• sparsity allows the solution to be stored in memory in a compact form
and to be easily used later.
• The sparser is the solution of a kernel algorithm, the less time and
memory.
kernel RLS(kernel recursive least squares)
• a stream of training examples：
where (𝑥1 , 𝑦1 ) ∈ ℝ𝑝 × ℝ denotes the current input-output pair.
• loss function of the KRLS algorithm：
where
and 𝑦𝑡 = (𝑦1 , … , 𝑦𝑡 )𝑇
• optimal weight vector：
where 𝛼𝑡 = (𝛼1 , … , 𝛼𝑡 )𝑇
• Then, the loss function of the KRLS can be rewritten as
,
𝐾𝑡 = Φ𝑡𝑇 Φ𝑡
The Kernel RLS Algorithm