C OMMONWEALTH OF MASSACHUSETTS
E XECUTIVE O FFICE OF ENVIRONMENTAL A FFAIRS
D EPARTMENT OF E NVIRONMENTAL P ROTECTION
Appendix C: 2004 Quality Control Report
per 40 CFR section 51.366(c)
Massachusetts Enhanced Emissions and Safety Test
Appendix C: 2004 Quality Control Report
TABLE OF CONTENTS
1.0
2.0
3.0
3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8
3.9
3.10
3.11
3.12
3.13
3.14
3.15
3.16
4.0
4.1
4.2
2004 PROGRAM CHANGES AFFECTING EQUIPMENT AUDIT RESULTS .................... 1
MASSDEP EQUIPMENT AUDIT PROCEDURE ................................................................ 2
2004 EQUIPMENT AUDIT RESULTS ................................................................................ 2
VISUAL INSPECTION ........................................................................................................... 4
GAS ANALYZER VISUAL INSPECTION ................................................................................... 5
WEATHER STATION AUDIT ................................................................................................. 5
LEAK CHECK ..................................................................................................................... 5
RAW TRANSPORT TIME CHECK........................................................................................... 6
GAS BENCH AUDIT ............................................................................................................ 7
GAS CAP TESTER ............................................................................................................. 8
INDUCTIVE RPM PICKUP.................................................................................................... 9
OBDII RPM AUDIT ........................................................................................................... 9
OBDII TESTER AUDIT .......................................................................................................10
VMAS VISUAL INSPECTION ...............................................................................................10
VMAS DILUTE O2 SENSOR ..............................................................................................11
VMAS FLOW ...................................................................................................................11
FUNCTIONAL DYNAMOMETER CHECK .................................................................................12
OVERALL AUDIT RESULTS .................................................................................................12
STATIONS SHUT DOWN AS A RESULT OF EQUIPMENT AUDITS...............................................12
REVISED AUDIT TOLERANCES ......................................................................................13
REVISED TOLERANCES FOR SECOND BENCH AUDITS ..........................................................13
REVISED TOLERANCES FOR VMAS DILUTE O2 AUDITS .......................................................14
2004 Annual Report: Attachment C – Quality Control Report
1.0 2004 Program Changes Affecting Equipment Audit Results
MassDEP’s 2004 equipment audit results were affected by five significant changes in the Program
that occurred during the year. These changes occurred n response to high equipment audit failure
rates for key emission measurement components reported in MassDEP’s 2002 and 2003 Annual
Reports to U.S. EPA:
 In order to focus quality control efforts, MassDEP defined “Critical Audit Failures,” which are
malfunctions that may substantially affect test accuracy. Components of these Critical Audit
Failures were defined as the gas bench, VMAS, gas cap tester, OBDII tester, and barometric
pressure.1
 Specific equipment reliability standards for the critical components (described in Section 4 of
this Report) were incorporated into the Program Contract via an amendment signed on June 1,
2004.
 To meet those standards, Agbar found it necessary to replace one vendor’s workstations. On
June 18, 2004, MassDEP stopped auditing the Workstations that were due to be replaced.
Once the workstations were replaced, MassDEP resumed audits for the replaced
Workstations. Agbar reported to MassDEP that all of the replacements were finished (on
schedule) by February 28, 2005.
 The Workstations that were not replaced were upgraded. Upgrades were completed for the
NOx measurement technology and the bar code scanners. The OBDII testers were updated to
be CAN compatible. Agbar reported to MassDEP that all upgrades were finished (on
schedule) by February 28, 2005.
 The contract amendment required Agbar to significantly increase its maintenance and
monitoring of workstations, and to provide early identification of needed adjustments and
repairs.
Audit results were also affected by changes in audit tolerances during 2004. The tolerances for the
second bench audit, (which is conducted after a failing first bench audit and a calibration), were
tightened in April 2004, increasing the chance of failing the second bench audit. The tolerances
for VMAS dilute O2 audit were modified as of November 17, 2004, to reflect the accuracy of the
VMAS O2 sensor. This modification made it easier to pass the VMAS dilute O2 audits, especially
for 8% O2. Failures for 8% VMAS dilute O2 are not critical audit failures. These tolerance changes
are described in more detail in Sections 3.1 and 3.2 below.
Due to these Program changes, 2004 was a year of transition for MassDEP’s equipment auditing
program. As noted in the sections that follow, equipment audit failure rates improved significantly
during the year, particularly for the components identified as “critical”. However, because the
equipment upgrades were not completed until February 2005, the full effect of the Program
changes implemented in 2004 will be reported in the 2005 Annual Report.
1
A Critical gas bench audit failure is when the bench fails its first bench audit, is re-calibrated and then
fails its second bench audit, immediately after the calibration. Critical VMAS failures include failures of
the VMAS flow audit or 15% dilute O2 audit. The Combined Critical Gas Bench/VMAS failure is when the
workstation has either a Critical Gas Bench or Critical VMAS failure.
1 of 14
2004 Annual Report: Attachment C – Quality Control Report
2.0 MassDEP Equipment Audit Procedure
MassDEP developed its equipment audit procedures based on recommendations from its
consultant, Sierra Research of Sacramento, CA (Sierra) and audit contractor, SGS TESTCOM, Inc.
(TESTCOM) of Ballston Spa, NY. Sierra developed general equipment audit guidelines for EPA
for ASM-type equipment used in transient test I&M programs 2. MassDEP’s audit procedure
mirrors EPA’s guidance document and includes many other checks that are unique to the
Massachusetts I&M Program. These additional checks are noted in section 2.0.
MassDEP’s equipment audit procedure includes the following parts:
Visual Inspection
Gas Analyzer Visual
Weather Station
Leak Check
Gas Bench
Gas Cap Tester
RPM pickups
OBDII Scanner
VMAS Visual
VMAS O2 Sensor
VMAS Flow
Visual check that all relevant equipment is connected, powered,
and ready to perform an inspection. Check for visual signs of
tampering.
Visual check of the gas analyzer including the condition of raw
sample system (probe, hose, and filters), on-board calibration
gases, and zero air gas generator.
Accuracy check of the workstation temperature, relative humidity,
and barometric pressure readings.
Functional check for leaks in the raw sample system and the
analyzer’s ability to detect leaks during its 24-hour calibration.
Accuracy check of the gas bench HC, CO, CO2, NOx, and O2
readings using 6 different audit gases.
Functional check of the workstation’s ability to distinguish between
passing and failing gas caps
Functional and accuracy check of the inductive and OBDII RPM
pickups at 700 and 2,500 RPM.
Visual check of the OBDII cable condition. Functional check of the
scanner’s ability to communicate with OBDII systems. Accuracy
check of the scanner’s ability to retrieve specific diagnostic trouble
codes (DTCs) and readiness monitor status.
Visual check for proper installation of VMAS and condition of hoses.
Accuracy check of the VMAS O2 sensor using three audit gases.
Accuracy check of VMAS flow measurement at three flow rates.
3.0 2004 Equipment Audit Results
MassDEP performed a total of 1,327 audits in 2004. 1095 different workstations (lanes) and 1089
different inspection stations were audited 3. 149 workstations were audited two times, 29
workstations were audited three times, 5 workstations were audited 4 times, 1 workstation was
“U.S. EPA Steady State and Transient Testing Equipment Audit Guidance,” July 2001
There were 8 stations that had 2 workstations (lanes) to audit and 1 station that had 3 workstations (lanes)
to audit. Four workstations (lanes) were audited at two different stations because workstations are
relocated as stations leave and others enroll in the Program.
2
3
2 of 14
2004 Annual Report: Attachment C – Quality Control Report
audited 5 times, and 1 workstation was audited 7 times. Re-audits occurred at stations failing
items deemed critical during the initial or follow-up audits.
As noted in Section 1.0 above, “critical audit failures”, malfunctions that may substantially affect
test accuracy, were defined in 2004 as: the gas bench, VMAS, gas cap tester, OBDII tester, and
barometric pressure. Of these components, the gas bench and VMAS have proven to be the most
difficult to maintain accuracy and are therefore monitored more closely. The following graph
presents the Critical failure rate for combined gas bench/VMAS failures4 in 2004.
Combined Critical Gas Bench/VMAS Audit Failure Rate by Month
- Includes Initial Audits and Re-Audits -
25%
Percent Failed
20%
15%
10%
5%
De
c04
No
v04
O
ct
-0
4
Se
p04
Au
g04
Ju
l-0
4
Ju
n04
ay
-0
4
M
Ap
r- 0
4
ar
-0
4
M
Fe
b04
Ja
n04
0%
Due to continued poor performance, one of the two inspection equipment vendors was removed
from the I&M Program in 2004. Starting June 18, 2004, this vendor’s equipment was no longer
used for performing exhaust emissions tests and was therefore no longer audited. The remaining
vendor replaced that equipment over a six-month period from September 2004 through February
2005. As seen in the above graph, the Combined Critical gas bench/ VMAS failure rate dropped
significantly starting in July 2004.
Table 1 below presents a breakdown of results for the individual audit parts and the overall audit
result. Although a total of 1,327 audits were performed, the number of audit parts tested (table
column “Tested”) was somewhat less than this because not all audits were complete audits. In the
case of re-audits, typically only the parts that failed the initial audit were retested. In other
instances, a particular part could not always be audited due to circumstances that are described in
the sections below. Also, the sum of the failure rates for the individual audit parts is greater than
100%. This occurs because a single audit may fail more than one audit part.
4
A Critical Gas Bench audit failure occurs when the bench fails its first bench audit, is re-calibrated and
then fails its second bench audit, immediately after the calibration. Critical VMAS failures include failures
of the VMAS flow audit or 15% dilute O2 audit. The Combined Critical Gas Bench/VMAS failure is when
the workstation has either a Critical Gas Bench or Critical VMAS failure.
3 of 14
2004 Annual Report: Attachment C – Quality Control Report
Table 1: Equipment Audit Results Summary
Audit Part
Pass
Visual Inspection
Gas Analyzer Visual Inspection
Weather Station
Leak Check
Raw Transport Time Check
1st Gas Bench Audit
2nd Gas Bench Audit
Gas Cap Tester
Inductive RPM Pickup
OBDII RPM Pickup
OBDII Tester
VMAS Visual Inspection
VMAS Dilute O2 Sensor
VMAS SAO Flow
Overall Audit Result*
Combined (Gas Bench + VMAS)**
Combined Critical Gas Bench/
VMAS***
2004 Audit Results
Fail
Tested Failure Rate
1,248
1,050
773
1,156
169
1,062
27
1,084
1,057
1,060
1,087
1,049
755
999
598
890
79
134
292
83
18
141
72
75
32
11
12
39
278
38
729
409
1,327
1,184
1,065
1,239
187
1,203
99
1,159
1,089
1,071
1,099
1,088
1,033
1,037
1,327
1299
6.0%
11.3%
27.4%
6.7%
9.6%
11.7%
72.7%
6.5%
2.9%
1.0%
1.1%
3.6%
26.9%
3.7%
54.9%
31.5%
1128
171
1299
13.3%
* To pass the overall audit, the workstation cannot fail any individual audit part.
** Combined results from 1st Gas Bench, VMAS Dilute O2, and VMAS Flow audits for each workstation audited.
*** A Critical gas bench audit failure occurs when a bench fails its first bench audit, is re-calibrated and then fails a
second bench audit, immediately after the calibration. Critical VMAS failure occurs when the VMAS fails the flow audit
or 15% dilute O2 audit. The Combined Critical Gas Bench/VMAS failure occurs when the workstation has a Critical
Gas Bench or VMAS failure.
One major goal of the 2004 Program changes was to reduce Combined Critical Gas Bench/VMAS
failures to a maximum of 10% on initial audits and 5% on re-audits following repairs. 2004 audit
results indicate that this goal can be achieved in 2005.
3.1 Visual Inspection
The visual inspection checks the type of workstation being audited (e.g. whether it is equipped with
a dynamometer or has a diesel opacity meter) and includes the following items:
MassDEP Visual Inspection Audit Items
Signs of tampering
Barcode scanner is operational
Gas cap tester is connected and calibration device NOT attached
Dynamometer (if equipped) is connected to workstation and powered
Vehicle cooling fan is accessible and functional
VMAS is connected to workstation and powered
4 of 14
Part of EPA’s
Audit Guidance?
√
√
√
√
2004 Annual Report: Attachment C – Quality Control Report
Workstation is connected to a dedicated phone line
Workstation clock shows proper time and date (within 5 minutes)
Diesel Opacity meter (if equipped) is connected to workstation and powered
In 2004, 79 of 1,327 workstations audited (6.0%) failed one or more of the visual inspection items.
3.2 Gas Analyzer Visual Inspection
The gas analyzer visual inspection checks the condition of the gas analyzer portion of the
workstation and its ancillary equipment and contains the following items:
DEP Gas Analyzer Visual Inspection Audit Items
There are no non-stock components in the sample system hose assembly
Auxiliary sample hose and probe are available for testing dual exhaust vehicles
Inlet filter housing and elements are present and appear functional
On-board calibration gas cylinders have an official BAR-97 label and bar code
Calibration gas cylinder concentrations match those stored on the analyzer
Workstation is equipped with a zero air generator (ZAG)
ZAG is connected to workstation and powered
ZAG inlet filter and filter element are present
Part of EPA’s
Audit Guidance?
None
In 2004, 134 of 1,184 workstations audited (11.3%) failed one or more of the gas analyzer visual
inspection items.
3.3 Weather Station Audit
The weather station audit checks the accuracy of the workstation’s weather station against a NISTtraceable calibrated portable audit weather station. The weather station audit contains the
following items and specifications:
MassDEP Weather Station Audit
Items
Barometric Pressure, in Hg
Temperature, deg F
Relative Humidity, %
DEP Pass/Fail Tolerance
± 3% of reading and ± 0.004 in Hg absolute
± 4 deg F, absolute
± 5% of reading and ± 3% absolute
Part of EPA’s
Audit Guidance?
√
√
√
In 2004, 292 of 1,065 workstations audited (27.4%) failed one or more of the weather station audit
items.
3.4 Leak Check
The workstation requires the inspector to perform a leak check of its raw exhaust sample system
as part of its 24-hour calibration procedure. This sample system consists of external components
such as a probe tip (that is inserted into the tailpipe), probe tip handle (that the sample flows
through), rubber hose, particle filter, and internal components such as the sample pump, solenoid
5 of 14
2004 Annual Report: Attachment C – Quality Control Report
valves, and assorted plumbing leading up to the gas bench. It is the station’s responsibility to
maintain the external sample system components so that they do not have any leaks.
The MassDEP leak check audit contains two checks for determining the condition of the raw
exhaust sample system and verifying the ability of the MASS99 system to self-detect leaks:
MassDEP Leak Check Audit Items
Part of EPA’s
Audit Guidance?
√
√
Leak check fails with probe uncapped
Leak check passes with probe capped
If the workstation cannot pass its leak check with the probe capped, the auditor attempts to make
repairs to obvious problems with the sample system (such as tightening a loose fitting) and repeats
the check. If the workstation still cannot pass its leak check with the probe capped, the auditor
replaces the probe tip, handle, and/or hose as needed to fix the problem and repeats the check. If
the workstation does not fail the leak check with the probe uncapped or pass the leak check with
the probe capped following repairs or parts replacement, a gas bench audit is not performed. In
these cases, the results are reported immediately to MassDEP personnel and a priority service visit
is arranged for the workstation.
In 2004, 83 of 1,239 workstations audited (6.7%) failed either the leak check capped or uncapped
test, a notable improvement over the failure rate of 47% reported in 2003. This improvement is the
result of two factors: implementing an aggressive inspection and maintenance program to track
leak-check related components and, in July 2004, switching the type of probe tip used, from a
flexible probe to a more rigid “needle” type probe.
3.5 Raw Transport Time Check
Only one of the equipment vendors, ESP, required a raw transport time check to be performed as
part of the 72-hour calibration routine performed by the inspector. The other vendor, SPX,
automatically determines the raw transport time for its sample system that cannot be changed by
the inspector. The raw-transport-time-check audit checks the raw transport time value (as
performed by the auditor) against the value stored in the workstation from the previous calibration
(as performed by the inspector). The raw transport time check has the following specifications:
MassDEP Raw Transport Time Check
MassDEP Pass/Fail Tolerance
Raw transport time, seconds
± 1 second absolute
Part of EPA’s
Audit Guidance?
no
In 2004, 18 of 187 ESP workstations audited (9.6%) failed the raw transport time check audit.
6 of 14
2004 Annual Report: Attachment C – Quality Control Report
3.6 Gas Bench Audit
The gas bench audit uses 6 different gases to check the ability of the workstation to accurately
measure the range of hydrocarbons (HC), carbon monoxide (CO), carbon dioxide (CO2), oxides of
nitrogen (NOx), and oxygen (O2) that may be present during an emissions test. The gas bench
audit has the following specifications:
Audit Gas
Blend
Component
Target
Concentration
Zero Air
1% oxygen
HC, ppm
CO, %
CO2, %
NO, ppm
O2, %
O2, %
0
0
0
0
20.9
1.0%
MassDEP audit
tolerance1st gas bench audit5
± 9 ppm absolute
± 0.04% absolute
± 0.6% absolute
± 26 ppm absolute
± 10% of bottle
± 0.2% absolute
BAR-97 Low
HC, ppm
200
± 6% of bottle
CO, %
0.50%
± 6% of bottle
CO2, %
6.0%
± 0.36% absolute
NO, ppm
300
± 28 ppm absolute
HC, ppm
CO, %
CO2, %
960
2.4%
3.6%
± 6% of bottle
± 6% of bottle
± 0.34% absolute
NO, ppm
HC, ppm
CO, %
CO2, %
900
1920
4.8%
7.2%
± 8% of bottle
± 6% of bottle
± 6% of bottle
± 6% of bottle
NO, ppm
HC, ppm
CO, %
CO2, %
NO, ppm
1800
3200
8.0%
12.0%
3000
± 8% of bottle
± 6% of bottle
± 6% of bottle
± 6% of bottle
± 8% of bottle
BAR-97 Mid 1
BAR-97 Mid 2
BAR-97 High
MassDEP audit
tolerance2nd gas bench audit6
± 9 ppm absolute
± 0.02% absolute
± 0.3% absolute
± 26 ppm absolute
± 6% of bottle
± 0.1% absolute and
± 1% of bottle
± 8 ppm absolute and
± 1% of bottle
± 0.02% absolute
and ± 1% of bottle
± 0.3% absolute and
± 1% of bottle
±25 ppm absolute
and ± 1% of bottle
± 4% of bottle
± 4% of bottle
± 0.3% abs and
± 1% of bottle
± 5% of bottle
± 4% of bottle
± 4% of bottle
± 0.3% abs and
± 1% of bottle
± 5% of bottle
± 4% of bottle
± 4% of bottle
± 4% of bottle
± 5% of bottle
Part of EPA’s
Audit Guidance?
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
If the workstation fails any one of the gases in the first gas bench audit, the auditor calibrates the
workstation and conducts the gas bench audit a second time.
In 2004, 141 of 1,203 workstations audited (11.7%) failed one or more of the gases during the first
gas bench audit. This is a significant improvement from 2003, when 21.5% failed one or more of
“Of bottle” indicates a percentage of the relevant audit gas bottle concentration.
As of 4/1/2004 second bench audit tolerances were made more stringent than the first bench audit
tolerances and more consistent with EPA audit guidance. See Section 3 for details about the changes in
tolerances.
5
6
7 of 14
2004 Annual Report: Attachment C – Quality Control Report
the gases during the first gas bench audit. In 2004, 72 of 99 workstations (72.7%) that failed the
first gas bench audit also failed the second gas bench audit following calibration7. Some
workstations could not pass the gas bench calibration after failing the first gas bench audit and
were automatically locked out from further testing. These workstations did not receive a second
gas bench audit.
3.7 Gas Cap Tester
The gas cap tester audit consists of checks that the station has all of the necessary gas cap
adapters and a gas cap calibration device, and checks the workstation’s ability to distinguish
between passing and failing gas caps. The gas cap tester audit contains the following items:
MassDEP Gas Cap Tester Audit Items
Part of EPA’s
Audit Guidance?
√
√
√
√
All gas cap adapters are available
Calibration device/caps are available
Audit “Pass” cap passes test
Audit “Fail” cap fails test
MassDEP uses two audit gas caps (one set to pass, the other set to fail) to audit the gas cap
tester. If the workstation does not produce the correct result for either of the audit gas caps, then it
fails the gas cap tester audit and the audit is repeated. If the audit fails either of the gas cap
retests, then the audit is repeated using the station’s calibration device. These additional steps
allow the gas cap audit to verify the initial results and determine if the station’s calibration device
was faulty.
In 2004, 75 of 1,159 workstations audited (6.5%) failed one or more of the items in the gas cap
tester audit. Despite efforts to fix this problem (which started in 2003), one of the equipment
vendors continued to have a noticeably higher failure rate than the other (14.4% versus 2.6%). As
mentioned previously, the equipment vendor with the higher audit failure rate was removed from
the Program, as of June 18, 2004. As seen in the graph below, the gas cap failure rate dropped
significantly after the vendor was removed.
7
Since the Workstations are calibrated frequently during normal operation, the majority of Workstations
that failed the first bench audit also fail the second bench audit that is conducted after a calibration.
8 of 14
2004 Annual Report: Attachment C – Quality Control Report
Failure Rate by Month for Gas Cap Audits
- Initial Audits and Re-Audits 25.0%
Percent Failed
20.0%
15.0%
10.0%
5.0%
De
c04
No
v04
O
ct
-0
4
Se
p04
Au
g04
Ju
l-0
4
Ju
n04
ay
-0
4
M
Ap
r- 0
4
ar
-0
4
M
Fe
b04
Ja
n04
0.0%
3.8 Inductive RPM Pickup
MassDEP uses an RPM signal generator to check the accuracy of the inductive RPM pickup, that
clamps around the sparkplug wire, at 700 and 2,500 RPM. The inductive RPM pickup audit
contains the following items:
MassDEP Inductive RPM pickup Audit
MassDEP Pass/Fail Tolerance
Items
RPM pickup is in good condition and connected to the workstation
Low Idle @ 700 RPM
± 3.3% of reading
High Idle @ 2,500 RPM
± 3.3% of reading
Part of EPA’s
Audit Guidance?
√
√
In 2004, 32 of 1,089 workstations audited (2.9%) failed one or more of the items in the inductive
RPM pickup audit. 10 of the 32 failures were due to poor condition of the pickup and/or cable
observed when inspecting of the RPM pickup and cable.
3.9 OBDII RPM Audit
The Massachusetts I&M Program uses the OBDII tester to record RPM from 1996 and newer
OBDII equipped vehicles if they receive a Two-Speed-Idle test. MassDEP uses an OBDII
simulator to generate a data stream to check the accuracy of the OBDII scanner at 700 and 2,500
RPM. The OBDII RPM audit contains the following items:
MassDEP OBDII RPM Audit Items
MassDEP Pass/Fail Tolerance
OBDII cable and connector are in good condition and connected to the workstation
Low Idle @ 700 RPM
± 10% of reading
High Idle @ 2,500 RPM
± 10% of reading
9 of 14
Part of EPA’s
Audit Guidance?
√
√
√
2004 Annual Report: Attachment C – Quality Control Report
In 2004, 11 of 1,071 workstations audited (1.0%) failed one or more of the items in the OBDII RPM
audit. 9 of the 11 failures were due to poor visible condition of the OBDII connector and not due to
inaccurate RPM readings.
3.10 OBDII Tester Audit
The OBDII Tester audit contains: 1) a check that the OBDII tester can communicate with a
vehicle’s OBDII system and 2) an accuracy check to verify that the OBDII tester correctly reads the
malfunction indicator light (MIL) status, diagnostic trouble codes (DTCs), and the status of all
readiness monitors. MassDEP uses an OBDII simulator to generate signals for MIL status, DTCs
and readiness monitor status that simulate the output of a vehicle’s OBDII system. The OBDII
tester audit consists of the following items:
MassDEP OBDII Tester Audit Items
Part of EPA’s
Audit Guidance?
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
Workstation communicates with OBDII simulator
MIL status
Misfire monitor status
Fuel System monitor status
Component monitor status
Catalyst monitor status
Heated Catalyst monitor status
Evaporative System monitor status
Secondary Air monitor status
A/C System monitor status
O2 Sensor monitor status
O2 Sensor Heater monitor status
EGR monitor status
DTC 1
DTC 2
DTC 3
DTC 4
DTC 5
DTC 6
In 2004, 12 of 1,099 workstations audited (1.1%) failed one or more of the items in the OBDII
Tester audit. All 12 failures were due to one vendor’s intermittent problem communicating with the
OBDII simulator and were not due to inaccuracies.
3.11 VMAS Visual Inspection
The VMAS visual inspection checks the condition of the VMAS and its ancillary equipment and
contains the following items:
MassDEP VMAS Visual Inspection Audit Items
VMAS blower is mounted correctly
Damage (cracks, leaks, loose fit) of the blower or VMAS tube
Vortex strut missing or severely damaged
10 of 14
Part of EPA’s
Audit Guidance?
None
2004 Annual Report: Attachment C – Quality Control Report
Thermistor bent or severely damaged
VMAS inlet hose is free from tears or leaks
VMAS inlet hose has an exhaust cone attached
VMAS inlet hose is between 5 and 20 feet long
Auxiliary hose, cone, and clamps are available for testing dual-exhaust vehicles
VMAS LEDs are blinking alternately each second (indicating no faults)
In 2004, 39 of 1,088 workstations audited (3.6%) failed one or more of the VMAS visual inspection
items. Note that there are relatively fewer VMAS audits than other audit parts because the VMAS
is typically omitted during re-audits if the VMAS passed the initial audit and because some VMAS
units were installed in locations that weren’t readily accessible to the auditors (e.g. hung out of
reach from ceilings or walls) and could not be audited safely. In some cases the visual portion of
the audit could still be performed, but the Dilute O2 sensor check and flow check could not due to
inaccessibility.
3.12 VMAS Dilute O2 Sensor
The VMAS dilute O2 sensor audit checks the accuracy of the VMAS O2 sensor using three audit
gases: 8% O2, 15% O2, and 20.8% O2 (zero air) and has the following specifications:
MassDEP VMAS Dilute O2 Sensor Audit
Items
8% O2 audit gas
15% O2 audit gas
20.8% O2 (zero air) audit gas
MassDEP Pass/Fail Tolerance8
± 5.3% of reading
± 5.3% of reading
± 5.3% of reading
Part of EPA’s
Audit Guidance?
√
√
In 2004, 278 of 1,033 workstations audited (26.9%) failed one or more of the VMAS dilute O2
sensor audit gases. As noted in Section 3.2, the pass/fail tolerances for the VMAS Dilute O2
sensor audit items were modified on November 17, 2004. The tolerance change contributed to the
improvement in the failure rate for this component compared to 2003, when 31.4% of audited
workstations failed one or more of the VMAS dilute O2 sensor audit gases.
3.13 VMAS Flow
The VMAS flow audit checks the accuracy of the VMAS flow sensor at three points: minimum flow
(~225 standard cubic feet per minute, or scfm), maximum flow (usually 400 to 450 scfm), and a
mid-point in between. MassDEP uses a smooth approach orifice (SAO) as a flow standard that is
compared the VMAS flow measurement. The VMAS flow audit has the following specifications:
MassDEP VMAS Flow Audit Items
MassDEP Pass/Fail Tolerance
Maximum Flow (~ 400 to 450 scfm)
Mid Flow
Minimum Flow (~ 225 scfm)
± 10% of reading
± 10% of reading
± 10% of reading
8
Part of EPA’s
Audit Guidance?
√
√
√
Tolerances for the VMAS dilute 02 audit changed as of 11/18/04 in order to better combine the Audit gas
and MASS99 Equipment tolerances. See Section 3 for details about the change in tolerances.
11 of 14
2004 Annual Report: Attachment C – Quality Control Report
In 2004, 38 of 1,037 workstations audited (3.7%) failed one or more of the VMAS flow points. This
is a significant improvement over the 11.2% failure rate in 2003.
3.14 Functional Dynamometer Check
The workstation requires a dynamometer coast-down check to be performed as part of its 72-hour
calibration procedure. If the coast-down check fails, the workstation automatically performs a test
to determine the parasitic losses in the dynamometer roll system. If these parasitic losses are
within manufacturer’s specifications, then a second coast-down check is performed using the new
parasitic loss values. If the parasitic loss values are not within manufacturer’s specifications or the
dynamometer fails it second coast-down check, then the dynamometer fails its calibration and the
workstation is automatically locked out from testing. Because the MASS99 system has this
extensive automated coast-down check and lockout mechanism, MassDEP’s audit procedure does
not specifically include performing a coast-down check for the workstation dynamometer.
3.15 Overall Audit Results
Overall, 729 of 1,327 workstation audits (54.9%) failed one or more of the 90 audit items described
in previous sections. This is an improvement over the 82.5% failure rate for 2003. 409 of 1,299
workstation audits (31.5%) failed either the first gas bench audit, VMAS Dilute O2 sensor, or VMAS
SAO Flow audits. In 2003, 38% of audits failed either the first bench audit or the VMAS audits.
For Combined Critical Gas Bench/ VMAS audit items, 171 of the 1,299 audited workstations
(13.3%) failed at least one of the Critical Bench or VMAS audit items.
3.16 Stations Shut Down as a Result of Equipment Audits
In 2004, 13 workstations of the 1,327 workstation audits (1.0%) were documented as shut down
during the course of the equipment audit. In these instances, the workstation was locked-out by its
own software because it could not pass one its required calibrations.
12 of 14
2004 Annual Report: Attachment C – Quality Control Report
4.0 Revised Audit Tolerances
Two revisions were made to audit tolerances in 2004:
1. starting on April 1, 2004, tolerances for the second bench audit were modified; and
2. starting on November 17, 2004, tolerances for the VMAS Dilute O2 audit were modified.
These tolerance changes are described in the following two sections.
4.1 Revised Tolerances for Second Bench Audits
As of April 1, 2004, audits were conducted with tighter tolerances for the second bench audit than
for the first bench audit. The gas bench audit tolerances, before and after the revision, are
presented in the following table. Revisions are highlighted in gray. The revised 2 nd gas bench
tolerances closely mirror those specified in EPA’s audit guidance.
MassDEP 2nd Bench Audit Tolerance
Audit Gas
Blend
Zero Air
1% oxygen
Component
HC, ppm
CO, %
CO2, %
NO, ppm
O2, %
O2, %
Target
Concentration
0
0
0
0
20.9
1.0%
1/1/2004 to 3/31/049
± 9 ppm absolute
± 0.04% absolute
± 0.6% absolute
± 26 ppm absolute
± 10% of bottle
± 0.2% absolute
BAR-97 Low
HC, ppm
200
± 6% of bottle
CO, %
0.50%
± 6% of bottle
CO2, %
6.0%
± 0.36% absolute
NO, ppm
300
± 28 ppm absolute
HC, ppm
CO, %
CO2, %
960
2.4%
3.6%
± 6% of bottle
± 6% of bottle
± 0.34% absolute
NO, ppm
HC, ppm
CO, %
CO2, %
900
1920
4.8%
7.2%
± 8% of bottle
± 6% of bottle
± 6% of bottle
± 6% of bottle
NO, ppm
HC, ppm
CO, %
CO2, %
NO, ppm
1800
3200
8.0%
12.0%
3000
± 8% of bottle
± 6% of bottle
± 6% of bottle
± 6% of bottle
± 8% of bottle
BAR-97 Mid 1
BAR-97 Mid 2
BAR-97 High
9
4/1/04 to 12/31/04
± 9 ppm absolute
± 0.02% absolute
± 0.3% absolute
± 26 ppm absolute
± 6% of bottle
± 0.1% absolute and
± 1% of bottle
± 8 ppm absolute and
± 1% of bottle
± 0.02% absolute and
± 1% of bottle
± 0.3% absolute and
± 1% of bottle
±25 ppm absolute and
± 1% of bottle
± 4% of bottle
± 4% of bottle
± 0.3% abs and
± 1% of bottle
± 5% of bottle
± 4% of bottle
± 4% of bottle
± 0.3% abs and
± 1% of bottle
± 5% of bottle
± 4% of bottle
± 4% of bottle
± 4% of bottle
± 5% of bottle
“Of bottle” indicates a percentage of the relevant audit gas bottle concentration.
13 of 14
2004 Annual Report: Attachment C – Quality Control Report
4.2
Revised Tolerances for VMAS Dilute O2 Audits
Following discussions with the manufacturer of the VMAS (Sensors, Inc.), tolerances for VMAS
dilute O2 audit were changed starting November 17, 2004. The accuracy of the VMAS O2 sensor
had been misinterpreted as 0.3% of reading instead of 0.3% absolute. The accuracy of the audit
gas remained unchanged at 5% of bottle concentration. The effect of this correction is that the
audit tolerance became slightly larger, most noticeable at the 8% concentration. Revisions are
highlighted in gray.
10
Audit Component
Zero Air O2, %
Target Concentration
20.9%
15% O2
15%
8% O2
8%
VMAS Dilute O2 Sensor Audit Tolerances
1/1/2004 to 11/16/0410
11/17/04 to 12/31/04
± 5% of bottle and
± 5.3% of bottle
± 0.3% absolute
± 5% of bottle and
± 5.3% of bottle
± 0.3% absolute
± 5% of bottle and
± 5.3% of bottle
± 0.3% absolute
“Of bottle” indicates a percentage of the relevant audit gas bottle concentration.
14 of 14