Calibration/Validation Guidelines
Minimum Travel Demand Model Calibration and Validation Guidelines for State of Tennessee Updated 2011
Calibration/Validation Guidelines
Tailor Fitting
Not too much,
Not too little,
JUST RIGHT!
Calibration/Validation Guidelines
Overview of Guidelines (pg. 3‐6)
Calibration/validation in the context of household surveys and transferable parameters
Role of reasonability checks
Role of sensitivity checks
Calibration/Validation Guidelines
Trip Generation Reasonability Checks (pg. 7‐9)
Trip Distribution Reasonability Checks (pg. 10‐13)
Trip Assignment
Where the “rubber hits the road” or “the synthetic modeled trips hits the link”
Traffic Assignment (pg. 13‐21)
Table
Check
5
P 15
Vol to Count Percent Error‐Functional Class
P 15
Screenlines
6
P 16
7
P 16
8
P 17
9
P 17
Percent Difference for Volume Groups
Fig 2
P 18
10
P 19
11
P 20
Cocf. Of Determination R2
P 20
Revised Percent Error by Design Class
Urban Area VMT by Facility Type
Percent Difference for VMT vs HPMS by Functional Class
Root Mean Square by Volume Group
Root Mean Square by Functional Group
Peak Hour Targets
Table 5 Volume to Count Ratios and Percent Error
2011
2003
Standards
Statistic
Acceptable
Standards
Preferable Statistic
Freeway Volume to Count
+/‐ 7%
+/‐ 6%
Freeway Volume to Count
Arterial Volume to Count
+/‐ 15%
+/‐ 10%
Arterial Volume to Count
Collector Volume to Count
+/‐ 25%
+/‐ 20%
Collector Volume to Count
One way/Frontage Road Volume to Count
+/‐ 25%
+/‐ 20%
One way/Frontage Road Volume to Count
•External model cordon lines should achieve +/‐ 1 percent
•Screenlines with greater than 70,000 AADT should achieve +/‐ 10 percent
•Screenlines with 35,000 AADT should achieve +/‐ 15 percent
•Screenlines with less than 35,000 AADT should achieve +/‐ 20 percent
•Cutlines +/‐ 15 percent
Acceptable
Preferable
Same FHWA/Michigan
Table 6
Volume to Count Ratios and Percent Error
Percent Difference Targets For Daily Volumes Groupings
2011
Average Annual Daily Traffic
<1,000
1,000 – 2,500
2,500 – 5,000
5,000 – 10,000
10,000 – 25,000
25,000 – 50,000
>50,000
Source: (8) (9) (10)
Desirable Percent Deviation
FHWA
200
100
50
25
20
15
10
Michigan
60
47
36
29
25
22
21
Percent Difference Targets For Daily Volumes Groupings
2003
Average Annual Daily Traffic
<1,000
1,000 – 2,500
2,500 – 5,000
5,000 – 10,000
10,000 – 25,000
25,000 – 50,000
>50,000
Desirable Percent Deviation
FHWA
200
100
50
25
20
15
10
Michigan
60
47
36
29
25
22
21
Table 7
Volume to Count Ratios and Percent Error
Revised Percent Error by Volume Group
2011
Revised Percent Error by Volume Group
2003
Standards
Statistic
Percent Error: LT 10000 Volume (2L road)
Percent Error: 10000 – 30000 (4L road)
Percent Error: 30000 – 50000 (6L road)
Percent Error: 50000 – 65000 (4‐6L freeway)
Percent Error: 65000 – 75000 (6L freeway)
Percent Error: GT 75000 (8+L freeway)
Acceptable
50%
Preferable
25%
30%
20%
25%
15%
20%
10%
15%
5%
10%
5%
No Comparable Standard
Table 8
Urban Area VMT by Facility Type Facility Type
Urban Area Population
2011
Small
Medium Large
(50‐
(200K‐ (>1M)
200K)
1M)
Freeways/Expressways
18‐23%
33‐38%
40%
Principal Arterials
37‐43%
27‐33%
27%
Minor Arterials
25‐28%
18‐22% 18‐22%
Collectors
12‐15%
8‐12%
8‐12%
Urban Area Population
2003
Small
Medium
Large
(50‐
(200K‐1M) (>1M)
200K)
Same
Same
Same
Same
Same
Same
Same
Same
Same
Same
Same
Same
Table 9 (new)
Percent‐Difference Targets for VMT by Functional Classification
Facility Type
Memphis Target
Ohio Target
Virginia Target Freeways
8‐12%
7%
10%
5%
Principal
Arterials
18‐22%
10%
15%
10%
Minor Arterials
27%
15%
20%
15%
Collectors
33%
15%
25%
20%
All Roads
‐‐
‐‐
10%
5%
Acceptable Recommended
Table 10
Percent Root Mean Square Error by Volume Group
2011 Standards
2003 Standards
Statistic
Acceptable Preferable
RMSE: LT 5,000 VPD
100% 45%
115%
RMSE: 5,000‐9,999 VPD
45% 35%
43.1%
RMSE: 10,000‐14,999 VPD
35% 27%
RMSE: 15,000‐19,999 VPD
30% 25%
RMSE: 20,000‐29,999 VPD
27% 15%
RMSE: 30,000‐49,999 VPD
25% 15%
RMSE: 50,000‐59,999 VPD
20% 10%
RMSE: 60,000+ VPD
19% 10%
19%
RMSE Area wide
45% 35%
30%
28%
25% (<40,000 vpd)
30% (>40,000 vpd)
Coefficient of Determination R2
2011
2003
.88
Same
Use of Scatter to Define Outliers
Same
Table 11
Root Mean Square Error (RMSE) By Functional Class(14)
2011 Small
2011 Large*
Regions
Regions
Freeways
20%
20%
None
Principle Arterials
30%
35%
None
Minor Arterials
40%
50%
None
Collectors
70%
60%
None
Functional Type
2003
Peak Hour Validation Targets
2011
2003
75% of freeway link volumes within +/‐20% acceptable
Same
50% of freeway link volumes within +/‐10% desirable
Same
75% of major arterial link volumes with 10,000 vehicles per day within +/‐30% acceptable
Same
50% of major arterial link volumes with 10,000 vehicles per day within +/‐15% desirable
Same
75% of minor arterial link volumes within +/‐
40% New
50% of minor arterial volumes within +/‐20% New
Sample Size – 2011 (same for 2003)
“The
regional agency should strive to obtain traffic
counts on 10 percent or more of the region wide
highway segments being analyzed, if resources allow.
This 10 percent goal also applies to the distribution
of counts in each functional classification (freeways
and principal arterials ̶ at a minimum). Validation for
groups of links in a screenline should include all
highway segments crossing the screenline.”
2003 Recommendations
2003
2011
For application in Tennessee, it is suggested the following tests be conducted
1. Percent difference in value for screenlines and link volumes Table 5/6
2. Percent difference in volume by classification
3. Coefficient of Determination R2
4. Root mean square for link volumes
Table 5
R2
Table 10
Recommendations
Include:
1. Percent Error by roadway facility ‐ Table 7
2. RMSQ overall
3. Urban Area VMT by Facility – Table 8
4. VMT to HPMS by Functional class – Table 9 (Memphis, Ohio, Virginia)
5. RMSQ by functional class – Table 11
6. Peak Hour Targets
7. Sample size statement
Table
5 P 15
P 15
Tennessee Calibration / Validation Applications
Knoxville
Chattanooga
Vol to Count Percent K
C
Error‐Functional Class
Screenlines
K
6 Percent Difference for K
P 16
Volume Groups
7 Revised Percent Error by *
P 16
Design Class
8 Urban Area VMT by Facility ‐
P 17
Type
9 Percent Difference for VMT K
P 17
vs HPMS by Functional Class
Fig 2 K
Cocf. Of Determination R2
P 18
10 Root Mean Square by K
P 19
Volume Group
11 Root Mean Square by K
P 20
Functional Class
P 20
Peak Hour Targets
‐
* New Criteria OFHWA mMichigan References
Nashville
Memphis
N
M
C
N
M
Cm
‐
M
*
*
*
C
‐
‐
‐
N
‐
C N
M
C
N
M
‐
‐
‐
‐
‐
‐
K ‐ Knoxville Regional Mobility Plan 2009 ‐ 34
C ‐ Appendix D Technical Memo 3 Volume 3 Travel Demand Model Documentation LRTP 2035
N ‐ Appendix E LRTP 2035 ‐ Base Year Model Development Note
M ‐ Technical Memo 9 Memphis Travel Demand Model
Table
2003 Status
Used # TPOs
5 F.C.
√
4
5 Screenline
√
4
6
√
3
7
New
̶
8
̶
1
9
̶
2
R2
√
4
10
√
4
11
̶
1
Peak Hour Targets
̶
Sample Size
̶
̶
̶
2011 Action
Willing to Jump on Board?
Questions/Comments?