Technology Update Seminar
THE COMPLETE REAL-TIME PCR WORKFLOW:
EXPERIMENTAL DESIGN, APPLICATIONS, DATA ANALYSIS
Performing RTqPCR experiments: Issues to consider
Experimental design
Ricardo Gonzalo Sanz
[email protected]
07/10/2015
1. Basic ideas of experimental design
2. Experimental Design conditions
1. Randomization
2. Replication
3. Local control
3. Experimental Design applied to RTqPCR
experiments
1. Sources of variation in RTqPCR experiment.
2. Breaking up a large study into several plates
4. Some references
1. Basic ideas of experimental design
2. Experimental Design conditions
1. Randomization
2. Replication
3. Local control
3. Experimental Design applied to RTqPCR
experiments
1. Sources of variation in RTqPCR experiment.
2. Breaking up a large study into several plates
4. Some references
1. Basic Ideas of Experimental Design
Experimental design should be a mandatory step in every experiment
•Is a structured, organized method for
determining the relationship between the
different factors affecting an experimental
process and the output of that process.
Sir Ronald A. Fisher
Father of modern Mathematical Statistics
and Developer of Experimental Design and
ANOVA
“To consult the statistician after an experiment is finished, is often
merely to ask him to conduct a post mortem examination. He can
perhaps say what the experiment died of. “
1. Basic Ideas of Experimental Design
Why are many life scientists so adverse to thinking about design?
It is common to think that time spent designing
experiments would be better spent actually
doing experiments
Some myths arise in biologists and statistical fields about that.
• “It does not matter how you collect your data, there will always be a statistical
‘fix’ that will allow you to analyse them”
• “If you collect lots of data something interesting will come out, and you’ll be able
to detect even very subtle effects”
• “Garbage In, Garbage Out”
• “In science, as in life: more haste, less speed”
1. Basic Ideas of Experimental Design
Components of an experiment?
• An experiment is characterized by the treatments and experimental units
to be used, the method treatments are assigned to units and the
responses are measured.
• Treatments, units, and assignment method specify the experimental
design.
What about analysis? Analysis is not part of design, but MUST be
consider during planning.
1. Basic Ideas of Experimental Design
Variability types that play in an experiment:
• Planned systematic variability: This is the difference in response
between treatments applied.
• Noise variability: random noise. Differences between two
consecutives measures. We cannot avoid that.
• Systematic variability not planned: Produce a systematic
variation in the results. A priori the reason is not known. It can
be avoided with the randomization and the local control.
1. Basic Ideas of Experimental Design
Important steps to define before begin the experiment:
•Establish the main objectives of the experiment. Avoid collateral
problems
•Identify all the noise sources: Treatment, experimental errors,…
•Allocate each experimental unit which each treatment
•Clarify the type of response expected in each treatment
•Determinate the number of individuals in each group
•Run a pilot study
1. Basic Ideas of Experimental Design
Important steps to define before begin the experiment:
•Establish the main objectives of the experiment. Avoid collateral
problems
•Identify all the noise sources: Treatment, experimental errors,…
•Allocate each experimental unit which each treatment
•Clarify the type of response expected in each treatment
•Determinate the number of individuals in each group
•Run a pilot study
1. Basic Ideas of Experimental Design
Important steps to define before begin the experiment:
•Establish the main objectives of the experiment. Avoid collateral
problems
•Identify all the noise sources: Treatment, experimental errors,…
•Allocate each experimental unit which each treatment
•Clarify the type of response expected in each treatment
•Determinate the number of individuals in each group
•Run a pilot study
1. Basic Ideas of Experimental Design
Important steps to define before begin the experiment:
•Establish the main objectives of the experiment. Avoid collateral
problems
•Identify all the noise sources: Treatment, experimental errors,…
•Allocate each experimental unit which each treatment
•Clarify the type of response expected in each treatment
•Determinate the number of individuals in each group
•Run a pilot study
•How the data will be statistically analysed.
1. Basic Ideas of Experimental Design
Why Experimental Design?
• We can design experiments to minimize any bias in the comparison
• We can design experiments so that the error in the comparison is small
• Very Important: We are in control of experiments, and having that
control allow us to make stronger inferences about the nature of
differences that we see in the experiment. We will be able to make
inferences about causation.
1. Basic Ideas of Experimental Design
Why Experimental Design?
•To save money and time
Less probability of bad results (mistakes,…)
 Only analyze the samples strictly necessary (less reagents, time,…)
Only collect the data you need to answer the objectives
•Ethical issues
Only use the animals needed
Treatment applied to the necessary animals
Careful when applying the treatments
1. Basic Ideas of Experimental Design
1. Basic Ideas of Experimental Design
A good experimental design must…
• Avoid systematic error: e.g. samples from one group processed with
instrument A and samples from the other group processed with instrument B.
• Be precise: try to maintain the random error as low as possible
• Allow estimation of error: enough replicates in each treatment
• Have broad validity: our experimental units should reflect the population
about which we wish to draw inference
1. Basic Ideas of Experimental Design
A good experimental design must…
• Avoid systematic error: e.g. samples from one group processed with
instrument A and samples from the other group processed with instrument B.
• Be precise: try to maintain the random error as low as possible
• Allow estimation of error: enough replicates in each treatment
• Have broad validity: our experimental units should reflect the population
about which we wish to draw inference
Randomization
Local Control
Replication
1. Basic ideas of experimental design
2. Experimental Design conditions
1. Randomization
2. Replication
3. Local control
3. Experimental Design applied to RTqPCR
experiments
1. Sources of variation in RTqPCR experiment.
2. Breaking up a large study into several plates
4. Some references
2. Experimental Design conditions
1. Randomization
Randomization
• The method for assigning treatments to units involves randomization (“any
individual experimental subject has the same chance as any other individual of
finding itself in each experimental group”)
• It is one of the most important elements of a well-designed experiment
• Made valid most of the statistical analysis usually performed
Haphazard is not randomized
e.g. 4 treatments to be assigned to 16 units
1.
Use sixteen identical slips of paper, 4 marked with A, 4 with B, and so on to D. Put the
slips of paper into a basket and mix them thoroughly. For each unit, we draw a slip of
paper from the basket and use the treatment marked on the slip
2.
Treatment A is assigned to the first four units we have encounter, treatment B to next
four units, and so on.
2. Experimental Design conditions
1. Randomization
Randomization
• The method for assigning treatments to units involves randomization (“any
individual experimental subject has the same chance as any other individual of
finding itself in each experimental group”)
• It is one of the most important elements of a well-designed experiment
• Made valid most of the statistical analysis usually performed
Haphazard is not randomized
e.g. 4 treatments to be assigned to 16 units
1.
Use sixteen identical slips of paper, 4 marked with A, 4 with B, and so on to D. Put the
slips of paper into a basket and mix them thoroughly. For each unit, we draw a slip of
paper from the basket and use the treatment marked on the slip
2.
Treatment A is assigned to the first four units we have encounter, treatment B to next
four units, and so on.
2. Experimental Design conditions.
1. Randomization
Randomization against confounding
The effect of the treatment cannot be distinguished from another
e.g. Consider a new drug treatment for coronary artery disease. 2 treatments to compare: this drug
treatment with bypass surgery. We have 100 patients in our pool of volunteers; they need to be
assigned to the two treatments. We then measure five-year survival as a response.
What sort of trouble can happen if we fail to randomize? Bypass surgery is a major operation, and
patients with severe disease may not be strong enough to survive the operation:
- stronger patients to surgery and the weaker patients to the drug therapy.
This confounds strength of the patient with treatment differences. The drug therapy would likely
have a lower survival rate because it is getting the weakest patients, even if the drug therapy is
every bit as good as the surgery.
2. Experimental Design conditions.
1. Randomization
Randomization against confounding
e.g. Cont.
Patient 1 to 100
Surgery
Drug therapy
2. Experimental Design conditions.
1. Randomization
Saying “randomly assign…” is sometimes easier to say than to do, especially in
complex designs.
Some tools may help
– Research Randomizer
http://www.randomizer.org/
– Interactive Statistical Calculation pages
http://statpages.org/
(look por “Experimental design”)
2. Experimental Design conditions.
2.Replication
Replication
• It is the basis of all experimental design
• It is the repetition of the basic experiment with another experimental units
How many
replicates I
need?
the more replicates we have, the more confident
we can be that differences between groups are real
and not simply due to chance effects
More replicates increase in time/money cost
• To be in mind:
1. To Know the variability of the technology used
2. Previous works with similar technology
3. Directly correlated with the precision of the experiment
2. Experimental Design conditions.
2. Replication
Replication
• Which type of replicates? Technical
Biological
• Statistical power: is the probability that a particular experiment will detect a
difference, assuming that there really is a difference to be detected.
1. Effect size
2. amount of random variation
3. number of replicates
There are computer programs to calculate it (you must provide effect sizes and
estimates of variation) (http://homepage.stat.uiowa.edu/~rlenth/Power/)
• Pooling samples: To pool or not to pool?
2. Experimental Design conditions.
3. Local control
Local control
•When the experimental units are not homogeneous (sex, smokers,…) or the
process to analyze them neither are (Kits lot numbers, batch,…)
We are not interested in to find out the
differences between the levels of the blocks
•Differences among blocks could hide differences among treatments.
•It transforms systematic variability not planned in planned systematic
variability.
2. Experimental Design conditions.
3. Local control
Local control
Sample Treatment
1
A
2
A
3
A
4
A
5
B
6
B
7
B
8
B
Sex
Male
Male
Male
Male
Female
Female
Female
Female
Batch
1
1
1
1
2
2
2
2
Sample Treatment
Sex
1
A
Male
2
A
Female
3
A
Male
4
A
Female
5
B
Male
6
B
Female
7
B
Male
8
B
Female
Batch
1
2
2
1
1
2
2
1
2. Experimental Design conditions.
3. Local control
Local control
Sample Treatment
1
A
2
A
3
A
4
A
5
B
6
B
7
B
8
B
Sex
Male
Male
Male
Male
Female
Female
Female
Female
Batch
1
1
1
1
2
2
2
2
Treatment are confounded
between sex and batch
Sample Treatment
Sex
1
A
Male
2
A
Female
3
A
Male
4
A
Female
5
B
Male
6
B
Female
7
B
Male
8
B
Female
Treatment is well balanced
Batch
1
2
2
1
1
2
2
1
2. Experimental Design conditions.
3. Local control
Local control or randomization?
• Local control assure you that differences are not due to blocks in the
sample
• Local control eliminate the noise due to differences among blocks
• Randomization is good for balance effects from variables not taken into
account from the beginning.
“Block what you can, randomize what you cannot” (George Box, 1978)
1. Basic ideas of experimental design
2. Experimental Design conditions
1. Randomization
2. Replication
3. Local control
3. Experimental Design applied to RTqPCR
experiments
1. Sources of variation in RTqPCR experiment.
2. Breaking up a large study into several plates
4. Some references
3. Experimental Design applied to RTqPCR experiments.
Typical hypothesis in a RTqPCR experiment:
 There is no difference in the expression of a gene between two or more subpopulations
(null hypothesis)
 There is difference between the subpopulations (alternative hypothesis)
Significant result from testing them depends on:
1. Treatment effect: the larger it becomes, the easier to resolve from the
confounding noise
2. Biological variability: unavoidable, but can minimised with biological
replicates
3. Technical noise: minimised with technical replicates, careful lab practice.
Kitchen et al. Statistical aspects of quantitave real-time PCR experiment design.
Methods 2010;50: 231-236
3. Experimental Design applied to RTqPCR experiments.
Main steps in a typical qPCR experiment
1. Experimental
design
2. Sample preparation
& Quality control
4. Data generation
3. Assay design and
validation
5. Data analysis
6. Statistics and
interpretation
3. Experimental Design applied to RTqPCR experiments.
2. Sample
preparation &
Quality control
Sampling
• Collection of samples
• Storage of samples prior to extraction
Nucleic Acid Extraction
• Method of extraction
• Presence of inhibitors
• Storage of RNA prior to RT Reaction
Nucleic Acid Quality and Quantification
• Check RNA Quality
• Good Quantification in order to balance RT Rxn.
3. Assay design
and validation
•
•
•
•
•
•
4. Data
generation
Assay validation
Choice of Chemistry
Choice of primers/probes
PCR efficiency
Dynamic Range of Assay
Choice of Endogenous Control
Reverse Transcription
• Selection of enzyme and priming strategy
• gDNA contamination?
• Presence of inhibitors?
Kubista, M Real-time qPCR Experimental Design Considerations (December 11, 2009)
3. Experimental Design applied to RTqPCR experiments.
1. Determine sources of variation in RTqPCR experiments
•
•
•
•
•
•
Assay validation
Choice of Chemistry
Choice of primers/probes
PCR efficiency
Dynamic Range of Assay
Choice of Endogenous Control
Tichopad et al. Design and Optimization of Reverse-Transcription Quantitative PCR Experiments.
Clinical Chemistry 2009;55: 1816-1823
3. Experimental Design applied to RTqPCR experiments.
1. Determine sources of variation in RTqPCR experiments
Liver Tissue
qPCR Assays: ACTB, IL1B, CASP3, FGF7
Blood
qPCR Assays : ACTB, IL1B, CASP3, IFNG
Cell Cultures
qPCR Assays : ACTB, H3F3A, BCL2, IL8
•
•
•
•
•
•
Assay validation
Choice of Chemistry
Choice of primers/probes
PCR efficiency
Dynamic Range of Assay
Choice of Endogenous Control
Single Cells: individual astrocytes from mouse brain
qPCR Assays: 18s
Tichopad et al. Design and Optimization of Reverse-Transcription Quantitative PCR Experiments.
Clinical Chemistry 2009;55: 1816-1823
3. Experimental Design applied to RTqPCR experiments.
1. Determine sources of variation in RTqPCR experiments
>90% of total variance
No dominant step
Sampling & RT 30%
variance
Sampling >90%
Tichopad et al. Design and Optimization of Reverse-Transcription Quantitative PCR Experiments.
Clinical Chemistry 2009;55: 1816-1823
3. Experimental Design applied to RTqPCR experiments.
1. Determine sources of variation in RTqPCR experiments
• σqPCR= 0.13 cycles
•σRT= 0.35 cycles
•σsampling (single cells)= 1.9 cycles
•σsampling (liver tissue )= 1.2 cycles
•σsampling (blood)= 0.05 cycles
Tichopad et al. Design and Optimization of Reverse-Transcription Quantitative PCR Experiments.
Clinical Chemistry 2009;55: 1816-1823
3. Experimental Design applied to RTqPCR experiments.
1. Determine sources of variation in RTqPCR experiments
Conclusions:
General
Recommendation
Upstream replicates are better than downstream replicates. Hence generally,
including more subjects is superior to any other replicates and should be
preferred as long as it is economically feasible.
Solid tissue
Several samples should be withdrawn from the same tissue and processed
separately (sampling replicates).
Other types or replicates are inferior
Blood
Producing RT replicates is superior to any other types of replicates.
Cell culture
The number of cell culture wells should be maximized prior to any other type of
replicates. Secondarily, increasing the number of RT replicates should be
considered.
Low copy
transcript
Replicates should be produced at the RT level rather than at any other.
Tichopad et al. Design and Optimization of Reverse-Transcription Quantitative PCR Experiments.
Clinical Chemistry 2009;55: 1816-1823
3. Experimental Design applied to RTqPCR experiments.
1. Determine sources of variation in RTqPCR experiments
Conclusions:
•qPCR variance will be higher in samples with CT’s >30
•Use of qPCR replicates has little justification
•The noise contributed by any step in the processing of a sample can be reduced
by performing replicates and by use of mean values in subsequent analyses.
•It is a good practice to run a pilot study. Subjects, samples and pre-processing
procedures are representative of those taken forward to the larger assay.
Tichopad et al. Design and Optimization of Reverse-Transcription Quantitative PCR Experiments.
Clinical Chemistry 2009;55: 1816-1823
3. Experimental Design applied to RTqPCR experiments.
2. Breaking up a large study into several plates
Breaking up a large study into several plates
Suppose a “big” study:
• 16 genes to analyze in 24 samples = 384 reactions
• in triplicate = 1152 wells
• 16 genes x 2 NTC = 32 + 1152 = 1184
• 3 endogens x 24 samples x 3 = 216 + 1184 = 1400 wells
• 1400 wells / 384 ≈ 4 plates (384)
• 1400 wells / 96 ≈ 15 plates (96)
3. Experimental Design applied to RTqPCR experiments.
2. Breaking up a large study into several plates
Two strategies in this situation:
1. Gene maximization
2. Sample maximization
Gene1 Gene2 Gene3 Gene4 Gene5 Gene6 Gene7 Gene8 Gene9 Gene10 Gene11 Gene12 Gene13 Gene14 Gene15 Gene16
Sample1
Sample2
Sample3
Sample4
Sample5
Sample6
Sample7
Sample8
Sample9
Sample10
Sample11
Sample12
Sample13
Sample14
Sample15
Sample16
Sample17
Sample18
Sample19
Sample20
Sample21
Sample22
Sample23
Sample24
3. Experimental Design applied to RTqPCR experiments.
2. Breaking up a large study into several plates
Two strategies in this situation:
1. Gene maximization: All genes analyzed for each sample are assayed in the
same plate
Gene1 Gene2 Gene3 Gene4 Gene5 Gene6 Gene7 Gene8 Gene9 Gene10 Gene11 Gene12 Gene13 Gene14 Gene15 Gene16
Sample1
Sample2
Sample3
Sample4
Sample5
Sample6
Sample7
Sample8
Sample9
Sample10
Sample11
Sample12
Sample13
Sample14
Sample15
Sample16
Sample17
Sample18
Sample19
Sample20
Sample21
Sample22
Sample23
Sample24
PLATE 1
PLATE 2
PLATE 3
PLATE 4
3. Experimental Design applied to RTqPCR experiments.
2. Breaking up a large study into several plates
Two strategies in this situation:
2. Sample maximization: All samples analyzed for each gene are assayed in the
same plate
Gene1 Gene2 Gene3 Gene4 Gene5 Gene6 Gene7 Gene8 Gene9 Gene10 Gene11 Gene12 Gene13 Gene14 Gene15 Gene16
Sample1
Sample2
Sample3
Sample4
Sample5
Sample6
Sample7
Sample8
Sample9
Sample10
Sample11
Sample12
Sample13
Sample14
Sample15
Sample16
Sample17
Sample18
Sample19
Sample20
Sample21
Sample22
Sample23
Sample24
PLATE 1
PLATE 2
PLATE 3
P
L
A
T
E
4
3. Experimental Design applied to RTqPCR experiments.
2. Breaking up a large study into several plates
Conclusions:
• In the gene maximization strategy, it is recommended that a few samples are
repeated in both runs (inter-run/plate calibrator samples in order to detect and
remove inter-run variation)
 Inter-run variability is mainly due to base-line correction and threshold settings
 Interplate calibrators should be very stable assays, run in triplicates, in fairly high
concentration (20<Cq<25=
• In general, the sample maximization strategy is to be preferred (absence of
sample related inter-run variation, easier to setup, fewer reactions).
Hellemans et al., Genome Biology, 2007
1. Basic ideas of experimental design
2. Experimental Design conditions
1. Randomization
2. Replication
3. Local control
3. Experimental Design applied to RTqPCR
experiments
1. Sources of variation in RTqPCR experiment.
2. Breaking up a large study into several plates
4. Some references
4. Some references
Some interesting links...
•http://www.3rs-reduction.co.uk/html/main_menu.html (An interactive short course on experimental
design for research scientists working with laboratory animals)
•http://www.tfrec.wsu.edu/anova/index.html (A field guide to Experimental Designs)
Some interesting books...
•A first course in design and analysis of experiments (Gary W. Oehlert, University of Minnesota)
•Experimental design for the life sciences (Graeme D. Ruxton, Nick Colegrave)
Some interesting articles…
• Hellemans et al. qBase relative quantification framework and software for management and automated analysis
of real-time quantitative PCR data, Genome Biology, 2007
• Tichopad
et al. Design and Optimization of Reverse-Transcription Quantitative PCR Experiments. Clinical
Chemistry 2009;55: 1816-1823
•Kitchen et al. Statistical aspects of quantitative real-time PCR experiment design. Methods 2010;50: 231-236