Development, implementation, and application
of an improved model performance evaluation
and diagnostics approach
Byeong-Uk Kim*, William Vizuete, and Harvey E. Jeffries
Department of Environmental Sciences & Engineering, University of North Carolina at Chapel Hill
*Georgia Department of Natural Resources
5th Annual CMAS Conference
October 17, 2006
Typical SIP modeling
Base case
emissions
Base case
Model
Performance
Evaluation
Future case
Future projected emissions
=
Base case
+
meteorology
Future case
emissions
Preset
controls
If attainment
demonstration
failed
Future
control case
Future control Proposed Preset
case emissions controls controls
More controls until passing
attainment demonstration
Model performance evaluation (MPE) is the process for assessing
the “reliability of model predictions.”
Issues with typical MPE practice
MPE for only (if not, mostly) ozone signals


No systematic evaluation about if models get right answers for
likely right reasons
No evaluation of winds with respect to chemical signals
“Waterfall” procedures and no explicit consideration of
the impact of model performance on policy choices


No further MPE for model inputs/outputs with respect to
proposed policy options once a MPE is done by following the
EPA guidance literally
Probable diagnostic evaluation after many ad hoc analyses
Over-dependence on statistical tests


No acceptance for partially useful modeling results
No systematic analysis for graphical measures
 Needs for investigation of possible causes of poor performance
The expected outcomes of MPE
Is the formulation of a model scientifically
acceptable in general? (i.e. what is the adequacy
and quality of model formulation for this use?)

Concerning if models simulate general causes
Does a model replicate the observations adequately?
(i.e. does it make predictions that match history?)

Examining if models get right answers for right reasons
Is a model usable for answering specific (e.g. policy)
questions? (i.e. does the model fulfill the designed
task?)

Assessing if models are usable for target purposes
Modified from the original questions in Beck, 2002
Protocol for Regulatory Ozone Modeling
Performance Tests (PROMPT)
PROMPT is a meta-protocol; it is a protocol for protocols.
Four phases of evaluation procedures




Does this model show or have all necessary components to produce the
phenomena that I can expect from the current best perceptual/conceptual
model? (Evaluation Phase One)
Can this model distinguish which precursor(s) to control for ozone reduction?
(Evaluation Phase Two)
How precisely can the model estimate control requirements? (Evaluation Phase
Three)
What are the possible biases in the prediction and the impact of biases on the
policy choice? (Evaluation Phase Four)
Performance measures will be examined in a “progressive” manner.

In later evaluation phases, more information will be investigated than earlier
phases of evaluation.
PROMPT emphasizes “day-by-day” and “site-by-site” performance
analyses and requires evaluators to examine meteorological inputs,
ozone, NOx and VOCs as well as geographical features.
Importance of consideration of
control options in MPE
Given two winds, A and B, control options for R can be
evaluated if the target emission source is the grey area.

Assuming the emission intensity in the grey area is
homogeneous in time and space
Illustration of PROMPT application
Houston-Galveston-Brazoria 8-hour Ozone SIP Modeling (“base1b”
was used for this example although “base1c” is the newest case)




Modeling period: 2000-08-16 ~ 2000-09-06
 Extensive observational data available through TexAQS 2000
campaign
 Almost same period of Houston-Galveston Mid-Course Review
(MCR) modeling for the 1-hour ozone (“base5b”)
In general, this episode shows a very Houston-specific ozone problem;
Transient High Ozone Events (THOEs) that are often characterized
by hourly ozone concentration changes more or equal than 40 ppb.
THOEs are often caused by epidemic highly reactive volatile organic
compounds (HRVOCs) emission events under ozone-conducive
conditions.
No official HRVOC emission event record available for 2000
 The possible existence of event emissions in 2000 can be inferred
from a study conducted by UT researchers.
Transient High Ozone Event in Houston
>10,000 lbs/hr ethylene release at La Porte, (6700
lbs between 11:00 AM and 11:25 AM) 3/27/2002
Modeling domain
Ship Channel
Galveston Bay
36 km
36 km
12 km
4 km
1 km
1 km
Modeling with improved model inputs
Issues in base5b (1-hour MCR):




Poor surface wind predictions
PBL height and vertical mixing
Over-predictions in NOx, CO, and HRVOCs at surface
monitors
Insufficient ozone formation
Major changes in inputs


Meteorology, Emissions, Chemistry, Boundary conditions
Yet, questionable 4-km grid resolution
Does the set of new inputs make base1b (8hour SIP) more “useful” for assisting decision
makers in choosing control options between
NOx control and VOCs control (or both)?
RegEvnt1
base5b (1km)
base5b with CMAQ
base5b
Older 1-h model
Newer 8-h model
Clinton (C35C)
Base5b Vertical Kv
Profile
1400
1300
1200
1100
1000
800
700
600
500
400
300
200
100
0
0
50
100
150
200
250
Kv
(020:029)
(021:029)
Base1b Vertical Kv
Profile
1000
900
800
12
9
7
5
3
350
600
500
400
300
200
100
0
0
50
100
150
200
250
Kv
(020:029)
(021:029)
2
3
4
5
6
(021:030)
7
8
9#NAME?
10 11 12
13
14
15
16
17
18
19
20
21
22
3100
3000
2900
2800
2700
2600
2500
2400
2300
2200
2100
2000
1900
1800
1700
1600
1500
1400
1300
1200
1100
1000
900
800
700
600
500
400
300
200
100
0
23
LST
camx420_pa_cb4.20000830.hgb8h.base1b.psito2n2.TCEQuh1_eta_tke.PA.ipr
Base1b (8-hr SIP)
Aggregated Box Heights
Detrained boxes
Entrained
16
12
9
7
5
3
350
0
(020:030)
400
1
Height, m
700
CAMx Layer (m)
16
(021:030)
3100
3000
2900
2800
2700
2600
2500
2400
2300
2200
2100
2000
1900
1800
1700
1600
1500
1400
1300
1200
1100
1000
900
800
700
600
500
400
300
200
100
300
0
Detrained boxes
Entrained
0
(020:030)
Aggregated Box Heights
Base5b (MCR)
Height, m
CAMx Layer (m)
900
3100
3000
2900
2800
2700
2600
2500
2400
2300
2200
2100
2000
1900
1800
1700
1600
1500
1400
1300
1200
1100
1000
900
800
700
600
500
400
300
200
100
300
0
1
400
2
3
4
5
6
7
8
9
10
11
12
LST
13
14
15
16
17
18
19
20
21
22
23
3100
3000
2900
2800
2700
2600
2500
2400
2300
2200
2100
2000
1900
1800
1700
1600
1500
1400
1300
1200
1100
1000
900
800
700
600
500
400
300
200
100
0
NOx emissions
Bush Airport
Only 6AM~5PM
Decrease ~4 ppb/h downtown/west Houston; increase perimeter counties
CO emissions
Decrease ~50 ppb in downtown/west Houston
ETH emissions
Only 7AM on 25, 29, and 31
Some increases in downtown/west Houston
NO2 Barchart
and time
series
Level 4 in model
CO
timeseries
Drops to model
boundary
condition at
night.
ISOP
timeseries
ETH
timeseries
O3
peak
8/25
base5b
Aloft NO2
and CO
base1b
Summary
of
Process
Analysis
Overall, OH distribution of reaction with NO , CO, CH4 ranges
2
41%~ 48%; very similar new OH radical source strength across
HG domain

This is somewhat low compared to other PA results in other areas.
A significant portion of the total OH reaction (=new OH x chain
length) is with NO2, CO, CH4, and other non-NO oxidizing paths.
The absolutely maximum amount of O3 that can be formed at
the four sites ranged from 127 ppb to 150 ppb minus the
emitted NO which ranged from 22 to 123 ppb, thus limiting
chemical ozone to values between 36 and 103 ppb of ozone.
Thus, the chemical production of O3 is inversely proportional to
the NOx at these four sites.
PAN is predicted to be very low at these sites, so is RNO3.
Major SIP-related question
•What are the implications from
insufficient radical source?
–The deficient radical sources result in
insensitivity to VOC precursors and
inhibited due to elevated levels of NOx.
–With current model configuration, VOC
controls will have little to no effect in
future control strategies.
Acknowledgement
The Houston Advanced Research Center
and the 8-hour ozone Coalition Group
Texas Commission on Environmental
Quality: Dr. Jim Smith for base1b and
base5b files
University of Houston: Dr. Daewon Byun
and Dr. Soontae Kim for Q20 files
Missing FORM?
Observational Evidence
Two potential sources of HCHO
Flares


98-99% combustion assumed, 1% to 2% emitted VOC composition
is assumed same as that fed to flare; rest assumed to be CO2. We
assumed that HCHO emitted was equal to VOC emitted.
“Flare case” - Assumed that VOCs fed to flares were partially
converted to HCHO and that an amount equal to another 1% was
emitted as HCHO. This added a total of 55, 58, and 59 tons on
25th, 30th and 31st. to 13 flares located mainly in the eastern part
of Houston
Mobile sources


New data (SWRI, 2005) on Heavy Duty Diesel show that HCHO is
23% of VOC and ethene is 18% of THC. HCHO was 5% of CO.
We added HCHO at 4% of low level CO
“Mobile case” - Based on AC obs, assumed that MV emissions did
not have enough HCHO. An appropriate factor appeared to be 4%
of CO. This added 167, 156, and 145 tons on 25, 30, and 31.
Delta
Ozone,
ppb
O8/25
13-16h
Flare case
Delta
Ozone,
ppb
O8/25
09-12h
Mobile case
Summary
Flare imputation caused >30 ppb increase in ozone
concentrations
CO ratio caused >18 ppb increase ozone
concentrations, more distributed
Increased peak ozone at almost every monitor
causing 4 monitors to match observations
~20% increase in new OH and ~30% in ozone
production
Still did not match observed HCHO.
XYL
2%
TOL
1%
OLE
2%
OH + (VOCs + CO + CH4 + NO2)
CRES
1%
25 Aug. 2000
Bayland Park
ISPD
4%
ISO
P
ETH
3%
NO2
25%
ETOH
0%
Total OH reacted:
105.29 ppb
MEOH
1%
VOC + CO + CH4:
75.9 ppb
PAR
8%
VOC:
52.63 ppb
CH4
4%
OPEN
0%
MGLY
1%
CO
19%
ALD2
10%
FORM
14%
TCEQbase1b
b1b.psito2n2