Interfaces EO data with Atmospheric and
Land Surface Model: Progress report 2
Liang Feng, Paul Palmer
Project Tasks
A. Reference OSSE system
B. High-resolution model
C. CO:CO2 ratios
Observing System Simulation Experiment Tool
I. Progress overview
Overall Aim: Developing a reference OSSE simulation for
community researchers to evaluate impacts of space-borne
atmospheric composition measurements on surface flux estimates.
Outputs:
1). OSSE framework with flexiable module and libraries.
Status: Done.
2). One complete example OSSE system for 1) simulating OCO-like
XCO2 observations; and 2) estimating regional CO2 fluxes by
assimilating XCO2 observations using an Ensemble Kalman Filter
(EnKF).
Status: Done .
3. Detailed documents. Done.
4. EOF and Visualization Tools. Done.
II. Download and Installation
 Codes and documents:
1) Packages for PyOSSE and EOF visualizations can be
downloaded from http://xweb.geos.ed.ac.uk/~lfeng
2) Installation guides has also been provided, covering how to:
• unzip archive;
• set python search path;
• build shared libraries;
• test codes.
3) documents and references can be found on-line or from local
directory.
 Data
• Being required mostly by the example OSSE system.
• Covering climatology for cloud, aerosol PDF , satellite orbit,
instrument averaging kernel, and GEOS-Chem model outputs.
• Being stored as ASCII text files, netCDF files and BPCH files in
www.esa-da.org/data/otool_data.tar.gz.
III. PyOSSE framework
 Data flow and directory structure
 Classes
 IO Modules
 Configuration files
 Data flow and directory structure
Observation
Simulation
Flux Forecasts
CTM
Surface flux
Ensemble
Obs operator
Obs
+random errors
Prior + error
(3-D concentrations)
Posteriori + error
Forecast
ETKF
EnKF assimilation
CTM
Ensemble forecasts
(3-D fields)
Obs operator
Model Obs
Ensemble
PyOSSE package has over 90 python/fortran modules,
stored in 8 subdirectories:
util (lib)
atmosphere (CTM outputs)
surface (flux ensemble)
observation (sample/convolve)
instrument (ORB, AVK. CLD …)
enkf (stv and etkf solvers)
example (OSSE for OCO)
 Classes:
We have defined about 40 python classes as containers for data and
functions of model objects
Transport
Atmosphere
Flux model
inventory
Met files
Surface flux
Top-down estimate
Measure
Observation
Example: class for state vector
class ens_stat_cl:
Class for state vector used to assimilate observations
Members: (total of 44)
------------------------------------------------------------1. attr_dict:<dict>: attributes
2. step_tag_lst:<list>: ID for run step
# variables for current assimilation window
3. wnd_mean_x0:<array, (wnd_nx)>: prior coefficient values for lag window
4. wnd_mean_x:<array, (wnd_nx)>: posterior coefficient values for lag window
5. wnd_dx:<array, (wnd_nx, wnd_ne)>: perturbations ensemble for coefficients
6. wnd_inc_m:<array, (wnd_ne>: increment matrix for ETKF data assimilation
7. wnd_xtm: <array, (wnd_nx, wnd_ne)>: transform matrix for ETKF data assimilation
8. wnd_xinc: <array, (wnd_nx)>: increment for ETKF data assimilation
# current analysis increments
9. inc_m:<array, (wnd_nx)>: increment matrix for current step
10. xtm: <array, (wnd_nx, wnd_ne)>: transform matrix for current step
11. xinc: <array, (wnd_nx)>: increment for ETKF data assimilation
Functions (total of 14)
==========================================================
1. __init__: initialization
6. construct_tcor_matrix: Create temporal correlation matrix
7. add_new_x_to_window: Add new apriori to assimilation window
10. do_assim: assimilate observations and update state
 Configurable IO modules
• PyOSSE relays on many pre-defined data sets such as satellite orbit,
instrument sensitivity, and cloud PDF etc.
• PyOSSE needs mechanism to exchange data with CTM.
• Data is usually given in different formats, and often needs preprocessing.
• Configurable IO modules are designed as bridges between host
classes and data files.
• These modules are easy to be re-configured or even be replaced.
Host class
IO module
Var-list
Var-dict
fdesc
fopen
fclose
fwrite
fread
Def-Var-List
Def-Var-dict
Create-fdesc
Openfile
Readfile
writefile
Closefile
Disk files
data
 Configuration files
We have developed a new type of text-based menu files to:
1. Numeric or string parameters as function inputs (traditional).
2. Object as parameters such as list, functions (new)
3. Object creater (new)
Example: vob_def.cfg for observation simulations allows users to choose
classes for satellite orbit, averaging kernel, cloud/AOD PDF , and sampling etc
#MENU (cloud)
#================================================
name|cloud
path|$DATAPATH$/clim_dat/
flnm|meancloud_frac_XMONTHX_landsea.dat
dict|__load:$MDPATH$.cloud_file_m:cld_varname_dict
fopen|__load:$MDPATH$.cloud_file_m:open_cloud_file
fread|__load:$MDPATH$.cloud_file_m:read_cloud_file
fclose|__load:$MDPATH$.cloud_file_m:close_cloud_file
fget|__load:$MDPATH$.cloud_file_m:get_cloud_data
keywords|__dict:Nil
fclass|__load:$MDPATH$.cloud_m:cloud_cl
#MEND
Menu term (sample) in vob_def.cfg
#MENU (data random sampling)
#=======================================================
name|sample
# cloud sampling
fcld_sample|__load:$MDPATH$.cloud_m:cloud_sample
fcld_penalty|None
fcld_keywords|__dict:Nil
# AOD
faod_sample|__load:$MDPATH$.aod_m:aod_sample
faod_uplimit|0.3
faod_keywords|__dict:Nil
# landcover
flc_sample|__load:$MDPATH$.landcover_m:sample_lc
flc_stype_lst|__load:$MDPATH$.ak_file_m:ak_lc_stype_lst
flc_keywords|__dict:Nil
#MEND
Menu term (averaging kernel ) in vob_def.cfg
#MENU (averaging kernel)
#=====================================================
name|ak
path|$DATAPATH$/oco/oco_ak/
flnm|aknorm_XSURFACEX.XEXTX
viewmode_dict|__load:$MDPATH$.ak_file_m:def_ak_viewmode_dict
stype_dict|__load:$MDPATH$.ak_file_m:def_ak_surf_type_dict
fopen|__load:$MDPATH$.ak_file_m:open_ak_file
fread|__load:$MDPATH$.ak_file_m:read_ak_file
fclose|__load:$MDPATH$.ak_file_m:close_ak_file
fget|__load:$MDPATH$.ak_file_m:get_ak_data
keywords|__dict:Nil
fclass|__load:$MDPATH$.ak_m:ak_cl
#MEND
……
IV. OSSE Example system
 Observation simulation
 Inversion
Observation simulation (example/obs_simulation):
1. Generate dummy observation
by using gen_dummy_obs.py
(and vod_def.cfg)
2. Generate model observation
by sampling actual model
outputs (gen_sat_obs.py and
sob_def.cfg)
 GEOS-Chem outputs can easily be replaced by user’s CTM simulations by
changing sob_def.cfg
#MENU (class for model profile)
#================================================
name|fc_prof
# model output
path|$DATAPATH$/gc_std/
# file name (empty to use defaults in )
flnm|'ts_satelliteXEXTX.XYYYYXXMMXXDDX.bpch'
fdiaginfo|'diaginfoXEXTX.dat'
ftracerinfo|'tracerinfoXEXTX.dat'
vname|single_profile
ext|'.ST001.EN0001-EN0002'
# output access
fopen|__load:ESA.util.gc_ts_file_m:open_ts_file
fread|__load:ESA.util.gc_ts_file_m:setup_daily_profile
fget|__load:ESA.util.gc_ts_file_m:get_mod_gp
fpres|__load:ESA.util.gc_ts_file_m:get_mod_pres
fclass|__load:ESA.atmosphere.ts_slice_m:gc_slice_cl
#MEND
Inversion (example/obs_simulation):
1. Settings:
default settings are defined in osse_def.cfg, configuring classes for state
vector, flux, perturbation ensemble, prior flux, observations, model forward
simulation, and model outputs sampling etc.
Example: CTM defined in osse_def.cfg
#MENU (CTM for single tracer run)
name|ctm
runpath|$FCRUN_ROOT$
datapath|$FCRUN_DATAPATH$
tra_st|1
tra_end|2
finput_gen|__load:ESA.example.enkf_oco.input_geos_gen:create_new_input_file
ext|ST001.EN0001-EN0002
runtype|3
runscript|rungeos.sh
fclass|__load:ESA.example.enkf_oco.run_geos_chem:geos_chem_cl
#MEND
2) Results:
We have a rich set of outputs for inversion results as well as diagnostic data
For example: Posterior fluxes
For example: Posterior model XCO2
V. EOF and visualization
The pyeof module
Visualization package
pyeof
• Pyeof defines a class eof_cl to find EOFs for temporal and spatial
data sets.
• It uses SVD technique to find eigenvalues and eigenstate for
covariance matrix.
• It is easy to use, and also provides functions for visualization of
EOFs and PCs
EOF1
Visualization
• We have developed a wxpython-based GUI application for quick
visualizations including 2D plots and animations.
• It also has a python shell to use a rich set of existing visualization
tools.
• We have also provided functions for common data processing.
VI Conclusion
 OSSE tool and visualization package are ready for public
release.
 This framework is simple and flexible, due to the adopted
algorithm (EnKF), and the way to implement it (classes,
configurable IO modules, and enhanced menu functions …).
 It has been successfully used to digest GOSAT observations.
 We are making further developments.
B. Hi-resolution modeling
I. Motivation
 Interpret and digest future satellite (like OCO-2) observation of
atmospheric CO2 concentrations:
 Model errors (transport and representation errors) have significant
impacts on top-down estimates of surface fluxes.
 Higher model resolution usually means better transport modelling, and
smaller representation errors.
 Provide benchmarks for coarser simulations or nested model
simulations.
 Itself is one by-product of our efforts to develop EnKF approach based
on nested GEOS-Chem transport model.
 In-cooperate other process-based models (such as plume model) into
GEOS-Chem (Gonzi et al, University of Edinburgh).
 Due to the finer temporal and spatial resolutions, It is more sensitive to
the details of ‘small scale’ processes.
II. Challenges:
 Availability of High-resolution emission inventories
• 1x1 monthly fossil fuel emissions (Oda et al).
• 1x1.25 3-hourly CASA biospheric CO2 exchanges (R. Kawa , J.
Collatz, and D. Liu)
• 1x1.25 daily biomass burning (R. Kawa , J. Collatz, and D. Liu)
• 4x5 monthly oceanic surface CO2 flux (Takahashi et al)
 Programming issues
1. Memory:
In particular, stack size is limited by most of fortran compiler.
Modifications have been made over GEOS-Chem v8.02 to limit the
use of common block and avoid static arrays in modules.
2. Parallelisation:
Mainly we have to reschedule parallelisation in tpcore modules,
and change functions in dao_mod.f .
3. Minor issues:
Mainly they are associated with diag outputs.
Results
Comparison with standard GEOS-Chem 4x5 simulations
CO2 /ppm (ML1-8)
CO2/ppm (ML1-8)
Range: 374 – 415 ppm
Range: 377 ppm – 403 ppm
Comparison with nested simulation
Global 0.5x0.666
CO2 /ppm (ML1-4)
Nested 0.5x0.666
CO2 /ppm (ML1-4)
Nested model being used in UK
GAUGE project
Comparison with GEOS-Chem drived by ECMWF winds
GEOS-5 winds
CO2 /ppm (ML1-8)
ECMWF winds
CO2 /ppm (ML1-8)
ECMWF Met fields may have stronger vertical transport across boundary layer.
Comparison to averaged aircraft profiles
--- GEOS-5 PBL
TABATINGO
N = 11
RIO BRANCO
N = 11
SANTAREM
N=7
ALTA FLORESTA
N = 11
Summary
 Native resolution simulation provides:
a) More realistic descriptions of observed variations
b) Benchmark for lower-resolution model
c) Framework for including detailed process models
 Our comparisons also reveal some possible model errors.
 We are now working on flux inversions for selected regions
• We have done some OSSEs for Astrium Project
• We are working on digesting ICOS data.
Spare slides
 Data Assimilation Module (Ensemble
Kalman Filter)
 Top-down Optimal surface flux estimation
Posteriori
Priori
gain
observation
model
x  x  K (y obs  Hx )
a
f
f
x: regional surface fluxes
yobs: measurements of atmospheric concentrations.
H: Jacobian (CTM model).
K=PfHT(HPfHT+R)-1 – Kalman gain matrix.
Pf: a priori uncertainty matrix
R: observation error matrix
 Ensemble Approach
Key features:
(Feng et al., 2009 ; 2011)
• Represent a-priori uncertainties by an ensemble of flux
perturbations.
𝐏𝑓 = ∆𝐗𝑓 (∆𝐗𝑓)T
• Use a lag windows to limit computational costs.
Any emission will only be constrained by observations within a following
limited time period. After that period, it is considered to be well-known.
• Project the ensemble of perturbations together with prior
estimates into the observation space using CTM.
• Use Ensemble Transform Kalman Filter (ETKF) to determine
posterior fluxes, and the associated uncertainties from digesting
observations.
A. State vector and ensemble representation of its
uncertainties
F ( x, y, t )  F ( x, y, t )   cl BF ( x, y, t )
0
l
l
𝐹 𝑥, 𝑦, 𝑡 : Surface fluxes
𝐹0 𝑥, 𝑦, 𝑡 : Prior estimates of surface fluxes
𝐵𝐹𝑙 𝑥, 𝑦, 𝑡 : Basis functions for pulse-like flux perturbations,
𝑐𝑙 :coefficients to be estimated:
State vector: 𝐱 = [𝑐1 , 𝑐2 , 𝑐3 , …, 𝑐𝑛 ];
Perturbations: 𝛅𝐱 = [𝛿𝑐1 , 𝛿𝑐2 ,𝛿𝑐3 , …, 𝛿𝑐𝑛 ]
Error covariance: 𝐏 = ∆𝐗 ∆𝐗 T
Full or partial representation: ∆𝐗 = [𝛿𝐱1 , 𝛿𝐱2 ,𝛿𝐱3 , …, 𝛿𝐱𝑚 ], 𝑚 ≤ 𝑛
We use region_map_c.py to defined as 8-day surface CO2
flux perturbations over 144 global regions.
We use gen_region_pb_flux_m.py to generate BFs from
biospheric GPP map and multi-layer maps for 144 regions.
B. Ensemble forecasts
• We change IO function in pb_flux_c.py to generate the
ensemble of flux perturbations according to the defined basis
functions and pre-defined uncertainties.
• Functions in ctm_config_c.py and ctm_restart_c.py are
override to support GEOS-Chem tagged runs to project flux
perturbation ensemble to the ensemble of atmospheric tracer
concentrations.
C. Projection of ensemble forecasts to the observation space
(ctm_world_c.py)
∆𝐘 = 𝐇 𝐱 + ∆𝐱 − 𝐇 𝐱
D. Inversion algorithm:Ensemble Transform Kalman Filter
(ETKF)
• A posteriori and the associated uncertainties are simultaneously
calculated using SVD (or sparse-matrix LU) technique.
𝐊 𝐞 = ∆𝐗 𝑓 (∆𝐘)𝑇 [∆𝐘(∆𝐘)𝑇 +R]-1
∆𝐗 𝑎 = ∆𝐗 𝑓 T
TTT=[1+ ∆𝐘R−1(∆𝐘)𝑇 ]-1
• Corresponding updates are made to the ensemble of model 3D
concentrations to retain the contributions of fluxes outside the
assimilation (time) windows (Feng et al, 2009)
 Algorithm testing: Comparison with the LSCE
4d-var system
In the comparison experiments, we have assimilated the same ACOS-GOSAT
B210 XCO2 retrievals over 2009 and 2010, but using different approaches:
EnKF (UoE) and 4d-var (LSCE)