A QC Strategy for Confirming
Out-of-Specification
Bioassay Results
Susan Robinson, Sr. QC Manager
28SEP2011
Where We're Going
2
1.
What is an Out-of-Specification (OOS) result?
2.
Why
y do we care?
3.
How do you pick a specification range?
4.
What does Quality Control (QC) need?
5.
Typical analytical method OOS investigation pathway.
6.
What about biological assays? A case study.
What is an Out-of-Specification Result?
Specification document describes acceptable product material
3
z
Product Description
¾
Manufacturer, bottling or vialing description, product
concentration, storage condition
z
Test categories
¾
Identify, potency, purity, safety
z
Test types
¾
Size exclusion chromatography, protein concentration, bioassay
z
Acceptance criteria for each test
¾
Conforms to reference
¾
High molecular weight species (HMW) ≤ 5.0%
¾
Report result
¾
50 - 200% Relative
R l ti potency
t
(RP)
What is an Out-of-Specification Result?
4
z
OOS Result is outside acceptance criteria range
¾
If Relative Potency acceptance criterion is 70-130%, a result of
69% is an OOS
z
Specification vs. Acceptance Criteria
¾
Specification is the document which contains all tests and
acceptance criteria
¾
Acceptance criterion is the acceptable range for each test
¾
Called an OOS result, not an OOAC result (out of acceptance
criteria)
Why Do We Care?
5
z
Patient safety
z
Regulatory
g
y commitment
z
Confirming product consistency
z
Business risk in paying for a lot that is later deemed unacceptable
z
Determining product expiration period
Determining Appropriate Spec Ranges
6
z
Process history: tolerance intervals used for data analysis
z
'Fit for use': downstream manufacturing
g
z
Test results from preclinical and clinical product lots
z
Analytical test method capability
z
Manufacturing process
z
Stability results: degradation pathways
z
Regulatory / compendial requirements
z
Criticality ranking of defined quality attributes
z
Ph i
Physicochemical
h i l and
d biological
bi l i l characterization
h
t i ti
z
Clinical phase appropriateness
QC Requirements
z
Consistent and meaningful results
z
Consistent and clear rules for interpreting
p
g results
Consistency is key!
Do the same thing every time in the same way
Exceptions
p
should be few,, far between and veryy well documented
7
Typical Analytical Method OOS Pathway
Potential OOS Result
Obtained
Open QC Investigation
No Root Cause
Determined
Open QA Investigation
Root Cause Determined
Suspect True OOS
Open QA Investigation
Root Cause Determined
Invalid Assay
No QA Investigation
Cond ct QA Investigation
Conduct
In estigation
Retest Sample (1X)
Investigation Confirms
OOS; No Retest
8
Retest Sample to Confirm
OOS
(e.g. 7X for Outlier Test)
Bioassays: Problem Children
From FDA Guidance for Industry: "Investigating Out-of-Specification
Test Results for Pharmaceutical Production," Oct 2006
http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/ucm070287.pdf
"Chemistry-based laboratory testing of biotechnology products that are
under the jurisdiction of CDER are within the scope of this guidance
guidance.
However, this guidance is not intended to address biological assays
(e.g., in vivo, immunoassays)."
So? What do you do?
9
Assumptions for This Bioassay OOS Pathway
All results are independent from each other
10
z
This must be determined by
y each company
p y
z
Different analysts
z
Different days
z
Different cell passages
Assumptions for This Bioassay OOS Pathway
You know the method variability around independent results
z
If multiple
p independent
p
results are used to determine a reportable
p
result, you may use the variability from the independent results.
z
If multiple non-independent results are used to determine a
reportable result, you must use the variability from the reportable
results.
For example:
11
z
3 plates from 3 days are averaged for a reportable result. They are
independent so you may use the method variability where each
result is 1 plate.
z
3 plates from 1 day are averaged for a reportable result. They are
dependent so you must use the method variability where each
result is 3 plates.
Product Variability vs. Method Variability
12
z
For each result, there is variability of result due to:
¾
Difference in lots (product or manufacturing variability)
¾
Variability of measurement method (test method)
z
Specification ranges must take both into account
Example Specification Range
Lower Specification
Upper Specification
Average Result of
Multiple Product Lots

- 3 SD
+ 3 SD
Specification range is 6 SD from
multiple lots of product
Variance descriptive
p
of product
p
13
Uncertainty Around Specification Limit
Lower Specification
- 1 Method SD
Upper Specification
+ 1 Method SD
Uncertainty range due to
measurement variability is 2
SD across specification line
Variance descriptive of test
method
14
Uncertainty Around Specification Limit
Lower Specification = 70%
Upper Specification
- 1 SD = - 7%
+1 SD = 7%
63%
77%
85%
66%
80%
71%
73%
All results are from the same sample. The
spread is a measure of measurement
variability. All results are equally likely.
Is the 73% any more "in spec" than the
66%?
15
Determining Bioassay Testing Pathway
Lower Specification
- 1 Method SD
+ 1 Method SD
Potential
Out of Spec
In Spec
Uncertain,
need more
info
16
Upper Specification
Result In Spec
z
No further testing required
For Example:
17
z
The proposed specification is 70 - 130% Relative Potency (RP)
z
Th method
The
th d SD as d
determined
t
i d iin validation
lid ti iis 10% RP
z
The range of "In Spec" is 81 - 119% RP
Determining Bioassay Testing Pathway
Lower Specification
- 1 Method SD
+ 1 Method SD
Potential
Out of Spec
In Spec
Uncertain,
need more
info
18
Upper Specification
Uncertain Result Pathway
L
Lower
S
Specification
ifi i
- 1 Method SD
+ 1 Method SD
95% Confidence Interval
(CI) around 1 Result
To decrease the uncertainty, reduce the CI around the result. This can be done by increasing the
N used to obtain a result.
We would like 95% CI around the result to equal 1 method standard deviation. To do that, use
the equation for large sample confidence interval, where:
CI = Z * (method SD/√N)
We find that CI ≈ 1 SD where Z = 1.96 (for 95% CI) when N = 4.
This assumes you know your method standard deviation. Use t-statistic instead of Z if you don’t
know SD very well.
19
Uncertain Result Pathway
L
Lower
S
Specification
ifi i
- 1 Method SD
+ 1 Method SD
95% Confidence Interval
(CI) around Initial Result

If Avg Result of N
= 4 is out of
spec, then report
as Confirmed
C fi
d
OOS; no further
testing
20

If Avg Result from
N = 4 is in spec,
then report
p as In
Spec


Result Uncertain
z
Further testing required to reduce 95% Confidence Interval around
result to equal 1 method SD.
z
Bring N of independent results to 4
z
Final result is average of 4
¾
¾
If final result is inside the spec range, the result is In Spec
If final
fi l resultlt is
i outside
t id the
th spec range, the
th resultlt is
i Confirmed
C fi
d OOS and
d no
further testing is performed
For Example,
Example where spec is 70-130%
70 130% and method SD is 10%:
21
z
"Uncertain" for the lower specification (70% RP) is 60 - 80% RP
z
"Uncertain"
Uncertain for the upper spec (130% RP) is 120 - 140% RP
z
Three additional assays must be performed if initial result is in
"uncertain" ranges
Determining Bioassay Testing Pathway
Lower Specification
- 1 Method SD
+ 1 Method SD
Potential
Out of Spec
In Spec
Uncertain,
need more
info
22
Upper Specification
Determining Bioassay Testing Pathway
Lower Specification
- 1 Method SD
Upper Specification
+ 1 Method SD
Potential
Out of Spec
FDA guidance allows an outlier test for analytical methods to confirm
the OOS or to determine that the initial result is invalid.
How many repeats needed for outlier test? Five or seven commonly
used for analytical methods. How many for bioassays?
23
Potential OOS Result Pathway
Lower Specification
Distance =
Number of repeats for outlier
test is based on distance of
initial result from theoretical
result.
Distance is determined using
method standard deviations as
unit of measure.
24
8 method
SD or SD8
Distance =
5 method
SD or SD5
Theoretical Potency
(100 %RP)
Potential OOS Result Pathway
Outlier Test
z
25
Total number of sample results (N, including initial result) is based
on distance in SD that the initial result is from theoretical result
(SDn)
# of SDn
Total N Needed
40
3
13
4
8
5
7
6
6
7
5
10
SD8
SD5
Potential OOS Result Pathway
To Know about Outlier Test
26
z
Table is based on having an outlier test at 80% power
z
80% power: the selected sample size (N) will allow you to conclude
that the outlier is an outlier 80% of the time
Potential OOS Result Pathway
Lower Specification
8 SD = 5 total results
More results (higher N) allows you
to distinguish finer differences
between results.
The further the initial result is from
the "target", the less precise you
need to be in order to see a
significant difference.
5 SD = 10 total results
27
Theoretical Potency
(100 %RP)
Result Potential OOS
For Example, where spec is 70-130%, method SD is 10% and
method range is 25 - 200%:
Outlier Test Sample Size Based on SD of 10
12
Potential OOS
10
Uncertain;
needs
more N
Uncertain
N
8
Potential OOS;
needs outlier test
In Spec
6
4
2
0
25
32
39
46
53
60
67
74
81
88
95 102 109 116 123 130 137 144 151 158 165 172 179 186 193 200
Result
28
Result Potential OOS
For Example, where spec is 70-130%, method SD is 5% and
method range is 25 - 200%:
Outlier Test Sample Size Based on SD of 5
12
Potential OOS
10
Uncertain;
needs
more N
Uncertain
N
8
Potential OOS;
needs outlier test
In Spec
6
4
2
0
25
32
39
46
53
60
67
74
81
88
95 102 109 116 123 130 137 144 151 158 165 172 179 186 193 200
Result
29
Potential OOS Result Pathway
30
z
Outlier test used to determine result as OOS or In Spec
z
Number of results needed in order to have statistically valid outlier
test is based on distance of initial result from theoretical value
z
If initial result is outlier, the initial result is determined to be invalid
and the final result is reported as the average of the remaining
results
z
If initial result is not outlier, initial result reported as Confirmed OOS
Where We Were
31
1.
OOS results are outside of defined acceptable range
2.
OOS results can directly
y impact
p
p
patient safety,
y, regulatory
g
y
commitments, business needs
3.
Specification ranges selected based on product characteristics,
method capability, regulatory requirements
4.
Quality Control needs consistent results and rules
5.
Typical analytical assay OOS pathway
6.
Biological assays OOS pathway
1.
In spec results
2
2.
Uncertain results
3.
Potential OOS results
Acknowledgements
32
z
Laureen Little, Principle Consultant, Quality Services
z
Todd Coffey,
y, Sr. Statistical Scientist II,, Seattle Genetics
z
Chuck Smith, Vice President of Quality, Seattle Genetics
z
Shawn Novick, Assoc. Director, Quality Control, Seattle Genetics
Any
yQ
Questions?