DARTTutorialSec'on5:
ComprehensiveFilteringTheory:Non-Iden'ty
Observa'onsandtheJointPhaseSpace
©UCAR
TheNa'onalCenterforAtmosphericResearchissponsoredbytheNa'onalScienceFounda'on.
Anyopinions,findingsandconclusionsorrecommenda'onsexpressedinthispublica'onarethose
oftheauthor(s)anddonotnecessarilyreflecttheviewsoftheNa'onalScienceFounda'on.
AMoreGeneralContextforFilteringwithGeophysicalModels
Dynamicalsystemgovernedby(stochas'c)DifferenceEqua'on:
dxt
( x t ,t ) d β t ,
= f ( x t ,t ) + G
t ≥ 0
(1)
(2)
Observa'onsatdiscrete'mes:
( xk,t k ) + v k ; yk = h
k = 1,2,...;
t k+1
> t k ≥ t 0
Observa'onalerrorwhitein'meandGaussian(nice,notessen'al).
(
v →
0,
R
N
k
k
)
(3)
(4)
(5)
Completehistoryofobserva'onsis:
{
Yτ = yl ;tl
≤ τ } Goal:Findprobabilitydistribu'onforstateat'met:
p (x,t | Yt )
DARTTutorialSec'on5:Slide2
AMoreGeneralContextforFilteringwithGeophysicalModels
Statebetweenobserva'on'mesobtainedfromDifferenceEqua'on.
Needtoupdatestategivennewobserva'ons:
p x,t
k | Yt k = p x,t
k | yk ,Yt k−1 (6)
(
) (
)
ApplyBayes’rule:
(
)
p x,t k | Ytk =
p(yk
k | Yt ) | x k ,Ytk−1 )p(x,t
k−1
(7)
(8)
(9)
p(yk | Ytk−1 )
Noiseiswhitein'me(3),so:
(
)
p yk | xk ,Ytk−1 = p ( yk | xk )
Integratenumeratortogetnormalizingdenominator:
k | Yt ) = p(y k | x)p(x,t
k |Y
t )dx
p(y
∫
k−1
k−1
DARTTutorialSec'on5:Slide3
AMoreGeneralContextforFilteringwithGeophysicalModels
Probabilitya`ernewobserva'on:
p yk | x p x,t k | Ytk−1
p x,t
k |Yt = k
p(yk | ξ )p(ξ ,t k | Yt )d ξ
(
)
(
∫
)
(
)
(10)
k−1
Exactlyanalogoustoearlierderiva'onexceptthatxandyarevectors.
EXCEPT,noguaranteewehavepriorsampleforeachobserva'on.
SO,let’smakesurewehavepriorsby‘extending’statevector.
DARTTutorialSec'on5:Slide4
AMoreGeneralContextforFilteringwithGeophysicalModels
Extendingthestatevectortojointstate-observa'onvector.
yk = h x k ,tk + vk ; k = 1,2,...;
t k+1
> t k ≥ t 0 (2)
(
)
Applyinghtoxatagiven'megivesexpectedvaluesofobserva'ons.
Getpriorsampleofobserva'onsbyapplyinghtoeachsampleof
statevectorx.
Letz = [x, y] bethecombinedvectorofstateandobserva'ons.
DARTTutorialSec'on5:Slide5
AMoreGeneralContextforFilteringwithGeophysicalModels
NOW,wehaveapriorforeachobserva'on:
p yk | z p
z,t
k | Yt k−1
p z,t k | Ytk =
(
)
(
∫ p(y
)
k
(
)
(10.ext)
| ξ )p(ξ ,t k | Ytk−1 )d ξ
DARTTutorialSec'on5:Slide6
DealingwithManyObserva'ons
Onemoreissue:dealingwithmanyobserva'onsinsetyk?
Letykbecomposedofssubsetsofobserva'ons:
1
2
s
y
=
y
,
y
,...,
y
k
k
k
k
Observa'onalerrorsforobs.insetiindependentofthoseinsetj.
s
i
p
y
|
z
=
p
y
Then:
k
k |z
i=1
Canrewrite(10.ext)asseriesofproductsandnormaliza'ons.
{
(
) ∏ (
}
)
DARTTutorialSec'on5:Slide7
DealingwithManyObserva'ons
Onemoreissue:dealingwithmanyobserva'onsinsetyk?
Implica'on:canassimilateobserva'onsubsetssequen'ally.
Ifsubsetsarescalar(individualobs.havemutuallyindependent
errordistribu'ons),canassimilateeachobserva'onsequen'ally.
Ifnot,havetwoop'ons:
1.  Repeateverythingabovewithmatrixalgebra.
2.  Dosingularvaluedecomposi'on;diagonalizeobs.errorcovariance.
Assimilateobserva'onssequen'allyinrotatedspace.
Rotateresultbacktooriginalspace.
Goodnews:Mostgeophysicalobs.haveindependenterrors!
DARTTutorialSec'on5:Slide8
HowanEnsembleFilterWorksforGeophysicalDataAssimila'on
1.  Usemodeltoadvanceensemble(3membershere)to'me
atwhichnextobserva'onbecomesavailable.
Ensemblestate
es'matea`erusing
previousobserva'on
(analysis)
Ensemblestate
at'meofnext
observa'on
(prior)
DARTTutorialSec'on5:Slide9
HowanEnsembleFilterWorksforGeophysicalDataAssimila'on
2.  Getpriorensemblesampleofobserva'on,y = h(x),by
applyingforwardoperatorhtoeachensemblemember.
Theory:observa'ons
frominstrumentswith
uncorrelatederrorscan
bedonesequen'ally.
DARTTutorialSec'on5:Slide10
HowanEnsembleFilterWorksforGeophysicalDataAssimila'on
3.  Getobservedvalueandobserva'onalerrordistribu'on
fromobservingsystem.
DARTTutorialSec'on5:Slide11
HowanEnsembleFilterWorksforGeophysicalDataAssimila'on
4.  Findtheincrementsforthepriorobserva'onensemble
(thisisascalarproblemforuncorrelatedobserva'onerrors).
Note:Differencebetween
variousensemblefiltermethods
isprimarilyinobserva'on
incrementcalcula'on.
DARTTutorialSec'on5:Slide12
HowanEnsembleFilterWorksforGeophysicalDataAssimila'on
5.  Useensemblesamplesofyandeachstatevariabletolinearly
regressobserva'onincrementsontostatevariableincrements.
Theory:impactofobserva'on
incrementsoneachstate
variablecanbehandled
independently!
DARTTutorialSec'on5:Slide13
HowanEnsembleFilterWorksforGeophysicalDataAssimila'on
6.  Whenallensemblemembersforeachstatevariableare
updated,thereisanewanalysis.Integrateto'meofnext
observa'on…
DARTTutorialSec'on5:Slide14
Non-Iden'tyObserva'onOperatorsinLorenz_63:
Tryobservingmean(x,y),mean(y,z),mean(z,x)using
obs_seq.out.averageasinputfile.
Sameerrorvarianceandfrequencyaspreviously.
Inmodels/lorenz_63/workeditinput.nml
&filter_nml
…
obs_sequence_in_name = "obs_seq.out.z”
Execute./filterprogramtoproduceanewassimila'on.
Lookattheerrorsta's'csand'meserieswithMatlab.
Recordtheerrorandspreadvaluesandcomparetoiden'tycase.
Errorismuchlarger!
Iden'tyobserva'onsremoveallregressionerror;
canbeverymisleading.
DARTTutorialSec'on5:Slide15
DARTTutorialIndextoSec'ons
1. 
2. 
3. 
4. 
5. 
6. 
7. 
8. 
9. 
10. 
11. 
12. 
13. 
14. 
15. 
16. 
17. 
18. 
19. 
20. 
21. 
22. 
23. 
24. 
25. 
FilteringForaOneVariableSystem
TheDARTDirectoryTree
DARTRunBmeControlandDocumentaBon
HowshouldobservaBonsofastatevariableimpactanunobservedstatevariable?
MulBvariateassimilaBon.
ComprehensiveFilteringTheory:Non-IdenBtyObservaBonsandtheJointPhaseSpace
OtherUpdatesforAnObservedVariable
SomeAddiBonalLow-OrderModels
DealingwithSamplingError
MoreonDealingwithError;InflaBon
RegressionandNonlinearEffects
CreaBngDARTExecutables
AdapBveInflaBon
HierarchicalGroupFiltersandLocalizaBon
QualityControl
DARTExperiments:ControlandDesign
DiagnosBcOutput
CreaBngObservaBonSequences
LostinPhaseSpace:TheChallengeofNotKnowingtheTruth
DART-CompliantModelsandMakingModelsCompliant
ModelParameterEsBmaBon
ObservaBonTypesandObservingSystemDesign
ParallelAlgorithmImplementaBon
Loca'onmoduledesign(notavailable)
Fixedlagsmoother(notavailable)
Asimple1DadvecBonmodel:TracerDataAssimilaBon
DARTTutorialSec'on5:Slide16