G(m)=d mathematical model
d
data
m
model
G
operator
d=G(mtrue)+ = dtrue + 
Forward problem: find d given m
Inverse problem (discrete parameter estimation):
find m given d
Discrete linear inverse problem: Gm=d
G(m)=d mathematical model
Discrete linear inverse problem: Gm=d
Method of Least Squares:
Minimize E=∑ ei2 = ∑ (diobs-dipre)2
dipre
o
Diobs
o
o
o
o
zi
ei
E=eTe=(d-Gm)T(d-Gm)
=∑ [di-∑Gijmj] [di-∑Gikmk]
i
j
k
=∑ ∑ mj mk ∑ Gij Gik -2∑ mj ∑ Gijdi + ∑ di di
j k
i
j
i
i
∂/∂mq [∑ ∑ mj mk ∑ Gij Gik ] = ∑ ∑ [jqmk+mjkq]∑GijGik
j k
i
j
k
i
= 2 ∑ mk ∑ Giq Gik
k
i
-2 ∂/∂mq [∑ mj ∑ Gijdi ] = -2∑jq∑Gijdi = -2∑ Giqdi
j
∂/∂mq [∑didi]=0
i
I
j
i
i
∂/∂mq = 0 = 2 ∑ mk ∑ Giq Gik - 2∑ Giqdi
k
i
i
In matrix notation:
GTGm - GTd = 0
mest = [GTG]-1GTd assuming [GTG]-1 exists
This is the least squares solution to Gm=d
Example of fitting a straight line
mest = [GTG]-1GTd assuming [GTG]-1 exists
1 1 …1
[GTG] =
1 x1
1 x2
.
x1 x2 .. xm
1 xm
m ∑ xi
[GTG]-1 =
∑ xi ∑ xi2
-1
m ∑ xi
=
∑ xi ∑ xi2
Example of fitting a straight line
mest = [GTG]-1GTd assuming [GTG]-1 exists
1 1 …1
[GTd] =
d1
d2
∑ di
=
.
x1 x2 .. xm
dm
m ∑ xi
-1
∑ di
[GTG]-1 GTd =
∑ xi ∑ xi2
∑ xi di
∑ xi di
The existence of the Least Squares Solution
mest = [GTG]-1GTd assuming [GTG]-1 exists
Consider the straight line problem with only 1 data point
?
?
m ∑ xi
o
?
[GTG]-1 =
-1
1 x1
-1
=
∑ xi ∑ xi2
x1 x12
The inverse of a matrix is proportional to the reciprocal of the
determinant of the matrix, i.e.,
[GTG]-1  1/(x12-x12), which is clearly singular,
and the formula for the least squares fails.
Classification of inverse problems:
Over-determined
Under-determined
Mixed-determined
Even-determined
Over-determined problems:
Too much information contained in Gm=d to possess an
exact solution … Least squares gives a ‘best’
approximate solution.
Even-determined problems:
Exactly enough information to determine the model
parameters. There is only one solution and it has zero
prediction error
Under-determined Problems:
Mixed-determined problems - non-zero prediction error
Purely underdetermined problems - zero prediction error
Purely Under-determined Problems:
# of parameters > # of equations
Possible to find more than 1 solution with 0 prediction
error (actually infinitely many)
To obtain a solution, we must add some information not
contained in Gm=d : a priori information
Example: Fitting a straight line through a single data
point, we may require that the line passes through the
origin
Common a priori assumption: Simple model solution
best. Measure of simplicity could be Euclidian length,
L=mTm = ∑ mi2
Purely Under-determined Problems:
Problem: Find the mest that minimizes L=mTm = ∑ mi2
subject to the constraint that e=d-Gm=0
(m)= L+∑ i ei = ∑ mi2 +∑ i [ di - ∑ Gijmj ]
∂(m)/∂mq= 2 ∑ mi ∂mi/∂mq-∑ i ∑ Gij∂mj /∂mq]
= 2mq - ∑ iGiq = 0
In matrix notation: 2m = GT (1),
along with Gm=d (2)
Inserting (1) into (2) we get
d=Gm=G[GT/2] , = 2[GGT]-1d and inserting into (1):
m = GT [GGT]-1d
- solution exist when purely
underdetermined
Mixed-determined problems
Over
Under
determined determined
Mixed
determined
1) Partition into overdetermined and underdetermined
parts, solve by LS and minimum norm - SVD (later)
2) Minimize some combination of the prediction error
and solution length for the unpartitioned model
(m)=E+2L=eTe+2mTm
mest=[GTG+2I]-1GTd - damped least squares
Mixed-determined problems
(m)=E+2L=eTe+2mTm
mest=[GTG+2I]-1GTd - damped least squares
 Regularization parameter
0th-order Tikhonov Regularization
||m||

||Gm-d||
Min ||m||2, ||Gm-d||2<
||m||

‘L-curves’
||Gm-d||
min ||Gm-d||2, ||m||2 < 
Other A Priori Info: Weighted Least Squares
Data weighting (weighted measures if prediction error)
E=eTWee
We is a weighting matrix, defining relative contribution of
each individual error to the total prediction error
(usually diagonal).
For example, for 5 observations, the 3rd may be twice
as accurately determined as the others:
Diag(We)=[1, 1, 2, 1, 1]T
Completely overdetermined problem:
mest=[GTWeG]-1GTWed
Other A Priori Info: Constrained Regression
di=m1+m2xi
Constraint: line must pass through (x’,d’): d’=m1+m2x’
Fm= [1 x’] [m1 m2]T = [d’]
Similar to the unconstrained solution (2.5) we get:
m1
est
m2est =
1
M
∑ xi
1
∑ xi
1
∑ xi2 x’
x’
0
-1
∑ di
∑ xidi
d’
o
o
o (x’,d’)
o
d
o
x
M ∑ xi
-1
∑ di
Unconstrained solution: [GTG]-1 GTd =
∑ xi ∑ xi2
∑ xi di
Other A Priori Info: Weighting model parameters
Instead of using minimum length as solution simplicity,
One may impose smoothness in the model:
-1 1
m1
-1 1
l=
.
m2
.
.
.
= Dm
.
.
-1 1
mN
D is the flatness matrix
L=lTl=[Dm]T[Dm]=mTDTDm=mTWmm, Wm=DTD
firsth-order Tikhonov Regularization - min||Gm-d||22+||Lm||22
Other A Priori Info: Weighting model parameters
Instead of using minimum length as solution simplicity,
One may impose smoothness in the model:
l=
1 -2 1
m1
1
m2
-2 1
.
.
.
.
.
= Dm
. .
.
1 -2 1
mN
D is the roughness matrix
L=lTl=[Dm]T[Dm]=mTDTDm=mTWmm, Wm=DTD
2nd-order Tikhonov Regularization- min||Gm-d||22+||Lm||22