System Identification
Lecture 12: ARX, Instrumental variables, Validation
Roy Smith
2016-12-6
12.1
ARX estimation
vpkq
ypkq
`
Gpθ, zq
Gpθ, zq “
upkq
Bpθ, zq
Apθ, zq
10 1
10 0
G
10 -1
10 -1
10 0
Freq: [rad/sec]
200
100
0
-100
-200
-300
-400
10 -2
G
10 -1
10 0
10 1
Freq: [rad/sec]
2016-12-6
12.2
ARX estimation
Experiment: K “ 31
1.5
u(k)
1
vpkq „ N p0, λq
0.5
λ “ 0.005
0
-0.5
-1
-1.5
-30
-20
-10
0
10
20
30
10
20
30
Sample index
5
y(k)
0
-5
-30
-20
-10
0
Sample index
2016-12-6
12.3
ARX example: Matlab calculations
um = ...
ym = ...
% input measurements
% output measurements
for k = 0:K-1,
vphi = [-ym(k-1) -ym(k-2) -ym(k-3) um(k-2) um(k-3)]’;
Rexpt = Rexpt + (1/N)*vphi*vphi’;
fexpt = fexpt + (1/N)*vphi*ym(k);
end
theta = Rexpt\fexpt;
Bvec = [0; 0; theta(4); theta(5)];
Avec = [1; theta(1); theta(2); theta(3)];
% least-squares fit
Ghat = tf(Bvec,Avec,-1);
For clarity the negative time initializations and negative index details are not shown.
2016-12-6
12.4
ARX estimation
Estimate: K “ 31
10 1
10 0
P
ARX (K=31)
10 -1
10 -1
10 0
Freq: [rad/sec]
200
Apθ, zq
a1
a2
a3
plant
-2.08
1.71
-0.61
est.
-1.81
1.37
-0.39
Bpθ, zq
b1
b2
plant
1.00
-1.10
est.
1.02
-0.86
100
0
-100
-200
P
ARX (K=31)
-300
-400
10 -2
10 -1
10 0
10 1
Freq: [rad/sec]
2016-12-6
12.5
ARX estimation
Estimate: K “ 127
10 1
10 0
10 -1
10 -1
10 0
Freq: [rad/sec]
200
Apθ, zq
a1
a2
a3
plant
-2.08
1.71
-0.61
est.
-2.00
1.62
-0.55
Bpθ, zq
b1
b2
plant
1.00
-1.10
est.
1.04
-1.05
100
0
-100
-200
-300
-400
10 -2
P
ARX (K=31)
ARX (K=63)
ARX (K=127)
10 -1
10 0
10 1
Freq: [rad/sec]
2016-12-6
12.6
ARX estimation
Statistics for 500 ARX estimation experiments
250
ARX: a 1 (N=127)
ARX: a 1 (N=63)
ARX: a 1 (N=31)
200
150
100
50
0
-3
-2.8
-2.6
-2.4
-2.2
-2
-1.8
-1.6
-1.4
-1.2
-1
2016-12-6
12.7
ARX estimation
Time-domain MSE for 500 ARX estimation experiments
250
ARX time MSE (N=127)
ARX time MSE (N=63)
ARX time MSE (N=31)
200
150
100
50
0
2016-12-6
0
0.5
1
1.5
2
2.5
12.8
Aside: Wilkinson’s polynomial
Nominal polynomial
wpxq “
20
ź
i“1
px ´ iq “ px ´ 1qpx ´ 2q . . . px ´ 20q.
The roots of wpxq are t1, 2, 3, ¨ ¨ ¨ , 20u .
Expanding gives,
wpxq “ x20 ´ 210x19 ` 20615x18 ´ 1256850x17 ` 53327946x16 ´ 1672280820x15
` 40171771630x14 ´ 756111184500x13 ` 11310276995381x12
´ 135585182899530x11 ` 1307535010540395x10
´ 10142299865511450x9 ` 63030812099294896x8
´ 311333643161390640x7 ` 1206647803780373360x6
´ 3599979517947607200x5 ` 8037811822645051776x4
´ 12870931245150988800x3 ` 13803759753640704000x2
´ 8752948036761600000x ` 2432902008176640000.
2016-12-6
12.9
Aside: Wilkinson’s polynomial
Perturbed polynomial
Modify the x19 coefficent:
Nominal
1
2
3
4
5
6
7
8
9
10, 11
12, 13
14, 15
16, 17
18, 19
20
2016-12-6
Perturbed
1.00000
2.00000
3.00000
4.00000
5.00000
6.00001
6.99970
8.00727
8.91725
10.09527 ˘
11.79363 ˘
13.99236 ˘
16.73074 ˘
19.50244 ˘
20.84691
´210 is replaced by ´210.0000001192
0.64350i
1.65233i
2.51883i
2.81262i
1.94033i
12.10
Instrumental variable estimation
A nonlinear estimation case
vpkq
ypkq
vpkq „ N p0, λq
upkq
Bpθ, zq
Apθ, zq
`
λ “ 0.005
P pzq
Qpzq
xpkq
Approach:
1. Estimate θ̂LS via linear regression.
2. Select Qpzq “ ÂLS pzq and P pzq “ B̂LS pzq.
3. Calculate θ̂IV .
2016-12-6
12.11
Instrumental variable estimation
Solution approach (using LS initialization)
vpkq
ypkq
1
Apθ, zq
xpkq
`
upkq
Bpθ, zq
P pzq
Qpzq
LS estimates used for filter: Qpzq “ ÂLS pzq and P pzq “ B̂LS pzq.
Instrumental variable method
Define instruments:
“
ζpkq “ ´xpk ´ 1q
θ̂IV “
2016-12-6
˜
...
´xpk ´ nq
upk ´ 1q
¸´1
N ´1
N ´1
1 ÿ
1 ÿ
T
ζpkqϕ pkq
ζpkqypkq.
N k“0
N k“0
...
upk ´ mq
‰
12.12
Instrumental variable estimation
Experiment: K “ 31
1.5
u(k)
1
vpkq „ N p0, λq
0.5
λ “ 0.005
0
-0.5
-1
-1.5
-30
-20
-10
0
10
20
30
10
20
30
Sample index
5
y(k)
x(k)
0
-5
-30
-20
-10
0
Sample index
2016-12-6
12.13
Instrumental variable estimation
Estimate: K “ 31
10 1
10 0
P
ARX (K=31)
IV (K=31)
10 -1
10 -1
10 0
Freq: [rad/sec]
200
Apθ, zq
a1
a2
a3
plant
-2.08
1.71
-0.61
est.
-2.16
1.82
-0.67
Bpθ, zq
b1
b2
plant
1.00
-1.10
est.
1.00
-1.16
100
0
-100
-200
-300
-400
10 -2
P
ARX (K=31)
IV (K=31)
10 -1
10 0
10 1
Freq: [rad/sec]
2016-12-6
12.14
Instrumental variable estimation
Estimate: K “ 127
10 1
10 0
10 -1
10 -1
10 0
Freq: [rad/sec]
200
Apθ, zq
a1
a2
a3
plant
-2.08
1.71
-0.61
est.
-2.10
1.74
-0.63
Bpθ, zq
b1
b2
plant
1.00
-1.10
est.
1.01
-1.10
P
ARX (K=31)
IV (K=31)
ARX (K=63)
IV (K=63)
ARX (K=127)
IV (K=127)
100
0
-100
-200
-300
-400
10 -2
10 -1
10 0
10 1
Freq: [rad/sec]
2016-12-6
12.15
Instrumental variable estimation
Statistics for 500 IV estimation experiments
300
IV: a 1 (N=127)
IV: a 1 (N=63)
IV: a 1 (N=31)
250
200
150
100
50
0
-3
2016-12-6
-2.8
-2.6
-2.4
-2.2
-2
-1.8
-1.6
-1.4
-1.2
-1
12.16
Instrumental variable estimation
Time-domain MSE for 500 IV estimation experiments
250
IV time MSE (N=127)
IV time MSE (N=63)
IV time MSE (N=31)
200
150
100
50
0
0
0.5
1
1.5
2
2.5
2016-12-6
12.17
ARX/IV comparison
Time-domain MSE for 500 ARX & IV estimation experiments
250
ARX time MSE (N=127)
IV time MSE (N=127)
200
150
100
50
0
2016-12-6
0
0.5
1
1.5
2
2.5
12.18
Validation
Model quality assessment
§
§
Identification experiments, Zexpt “ typkq, upkqu,
Model fitting:
§
§
§
§
§
§
§
§
Least-squares (ARX);
Pseudo-linear regressions;
Instrumental variable methods;
Nonlinear optimization.
Validation experiments, Zval “ typkq, upkqu,
Compare models on validation data:
§
k “ 0, . . . , K ´ 1.
k “ 0, . . . , Kval ´ 1.
Mean-square error;
Residual characteristics.
Select “best” model.
2016-12-6
12.19
Example
Experiment: K “ 127
1.5
1
0.5
u(k)
0
-0.5
-1
-1.5
-20
0
20
40
60
80
100
120
80
100
120
Sample index
5
0
y(k)
-5
-20
0
20
40
60
Sample index
2016-12-6
12.20
ARX and IV models
Model structure parametrisation
na “ number of denominator coefficients
nb “ number of numerator coefficients
nk “ input delay
`
˘
b1 z na ´1 ` b2 z na ´2 ` ¨ ¨ ¨ ` bnb z na ´nb z ´nk
Gpzq “
z na ` a1 z na ´1 ` a2 z na ´2 ` ¨ ¨ ¨ ` ana
`
˘
b1 z ´1 ` b2 z ´2 ` ¨ ¨ ¨ ` bnb z ´nb z ´nk
“
1 ` a1 z ´1 ` a2 z ´2 ` ¨ ¨ ¨ ` ana z ´na
2016-12-6
12.21
ARX and IV models
Model structures
2016-12-6
Number
order
na
nb
nk
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
2
2
2
3
3
3
3
3
4
4
5
6
7
8
9
10
1
1
1
1
1
2
2
3
4
5
6
7
8
9
10
0
1
0
1
2
0
1
1
0
0
0
0
0
0
0
3
4
5
6
7
8
9
10
12.22
ARX and IV models
Data fitting errors
10 2
IV
ARX
Fit error
10 1
10 0
10 -1
0
2
4
6
8
10
12
14
16
Model number
2016-12-6
12.23
ARX and IV models
Data fitting and validation errors
10 2
IV
ARX
IV (val)
ARX (val)
Fit error
10 1
10 0
0
2
4
6
8
10
12
14
16
Model number
2016-12-6
12.24
ARX and IV models
Data fitting and validation errors
0.6
IV
ARX
IV (val)
ARX (val)
0.55
0.5
Fit error
0.45
0.4
0.35
0.3
0
2
4
6
8
10
12
14
16
Model number
2016-12-6
12.25
Identification approach
Typical workflow
§
Assess basic characteristics, signal levels, noise, drift, saturation, etc.
§
Build a “typical” simulation model for your system.
Develop experimental procedures (on the simulation system):
§
§
§
§
§
Test identification algorithms’ capabilities (on the simulation system).
§
§
§
2016-12-6
Sample-time selection;
Input signal design;
Data lengths.
Prefiltering;
Averaging;
Transfer function smoothing (γ selection).
§
Run experiments.
§
Fit a selection of models (various structures).
§
Run validation experiments.
§
Assess models and select the best model.
12.26
Bibliography
Prediction error minimization
Lennart Ljung, System Identification;Theory for the User, 2nd Ed., Prentice-Hall,
1999, [sections 7.1, 7.2 & 7.3].
Parameter estimation statistics
Lennart Ljung, System Identification;Theory for the User, 2nd Ed., Prentice-Hall,
1999, [section 7.4].
Correlation and instrumental variable methods
Lennart Ljung, System Identification;Theory for the User, 2nd Ed., Prentice-Hall,
1999, [sections 7.5 & 7.6].
2016-12-6
12.27