Grant-writing for Implementation
Sciences Research
• Mary Beth Weber
• K.M. Venkat Narayan
– Emory Global Diabetes Research Center
Translational research transforms currently
available knowledge into useful measures for
everyday clinical and public health practice.
Translational research aims to assess
implementation of standards of care,
understand the barriers to their
implementation, and intervene throughout all
levels of health care delivery and public health
to improve quality of care and health outcomes,
including quality of life.
Narayan et al. Ann Intern Med, 2004
DISTRIBUTION
AVAILABILITY
EFFICIENCY
EFFECTIVENESS
EFFICACY
FUNDAMENTAL
RESEARCH
DETSKY, 1990
Presentation overview
1. Study aims and objectives
2. Background
3. Research plan
4. Sustainability
5. Budget
1. Study Aims and Objectives
Aim: To evaluate the effectiveness, costAims
and objectives
effectiveness,
and sustainability of a communitybased diabetes prevention program in Chennai,
India using a randomized controlled trial, with
700 people with pre-diabetes randomized to
either standard lifestyle advice or a culturally
specific, low-cost, intensive lifestyle intervention
– Objective 1: To evaluate effectiveness of the
lifestyle intervention by assessing between
group changes in the following:
• Primary Outcome: incidence of T2DM
• Secondary Outcomes: body weight, percent
body fat, body mass index, waist-to-hip
ratio, fasting glucose, blood pressure,
plasma lipids, activity, and diet
Aim: To evaluate the effectiveness, costeffectiveness, and sustainability of a communitybased diabetes prevention program in Chennai,
India using a randomized controlled trial, with
700 people with pre-diabetes randomized to
either standard lifestyle advice or a culturally
specific, low-cost, intensive lifestyle intervention
– Objective 1: To evaluate effectiveness of the
Aim:
Clearly
states theby
point
of thebetween
study,
lifestyle
intervention
assessing
including
details about the
group
changesimportant
in the following:
research. The aim can also be stated as
• Primary
Outcome:
of T2DM
a question
or as a incidence
hypothesis.
• Secondary Outcomes: body weight, percent
body fat, body mass index, waist-to-hip
ratio, fasting glucose, blood pressure,
plasma lipids, activity, and diet
Aim: To evaluate the effectiveness, costeffectiveness, and sustainability of a communitybased diabetes prevention program in Chennai,
India using a randomized controlled trial, with
700 people with pre-diabetes randomized to
either standard lifestyle advice or a culturally
specific, low-cost, intensive lifestyle intervention
– Objective 1: To evaluate effectiveness of the
lifestyle intervention by assessing between
group changes in the following:
• Primary Outcome: incidence of T2DM
• Secondary Outcomes: body weight, percent
body fat, body mass index, waist-to-hip
ratio, fasting glucose, blood pressure,
plasma lipids, activity, and diet
Aim: To evaluate the effectiveness, costeffectiveness, and sustainability of a communitybased diabetes prevention program in Chennai,
India using a randomized controlled trial, with
700 people with pre-diabetes randomized to
either standard lifestyle advice or a culturally
specific, low-cost, intensive lifestyle intervention
– Objective 1: To evaluate effectiveness of the
lifestyle intervention by assessing between
group changes in the following:
• Primary Outcome: incidence of T2DM
• Secondary Outcomes: body weight, percent
body fat, body mass index, waist-to-hip
ratio, fasting glucose, blood pressure,
plasma lipids, activity, and diet
Aim: To evaluate the effectiveness, costeffectiveness, and sustainability of a communitybased diabetes prevention program in Chennai,
India using a randomized controlled trial, with
700 people with pre-diabetes randomized to
either standard lifestyle advice or a culturally
specific, low-cost, intensive lifestyle intervention
– Objective 1: To evaluate effectiveness of the
lifestyle intervention by assessing between
group changes in the following:
• Primary Outcome: incidence of T2DM
• Secondary Outcomes: body weight, percent
body fat, body mass index, waist-to-hip
ratio, fasting glucose, blood pressure,
plasma lipids, activity, and diet
Aim: To evaluate the effectiveness, costeffectiveness, and sustainability of a communitybased diabetes prevention program in Chennai,
India using a randomized controlled trial, with
700 people with pre-diabetes randomized to
either standard lifestyle advice or a culturally
specific, low-cost, intensive lifestyle intervention
– Objective 1: To evaluate effectiveness of the
lifestyle intervention by assessing between
group changes in the following:
• Primary Outcome: incidence of T2DM
• Secondary Outcomes: body weight, percent
body fat, body mass index, waist-to-hip
ratio, fasting glucose, blood pressure,
plasma lipids, activity, and diet
Aim: To evaluate the effectiveness, costeffectiveness, and sustainability of a communitybased diabetes prevention program in Chennai,
Objective: Describes how the investigators will
India using a randomized controlled trial, with
answer the research question
700 people with pre-diabetes randomized to
either standard lifestyle advice or a culturally
specific, low-cost, intensive lifestyle intervention
– Objective 1: To evaluate effectiveness of the
lifestyle intervention by assessing between
group changes in the following:
• Primary Outcome: incidence of T2DM
• Secondary Outcomes: body weight, percent
body fat, body mass index, waist-to-hip
ratio, fasting glucose, blood pressure,
plasma lipids, activity, and diet
Aims and objectives
• Aim 1:
Can peer education and support groups
be used to influence lifestyle choices and
prevent obesity in men living in Karachi?
Aim example
• Aim 1:
Can peer education and support groups
be used to influence lifestyle choices and
prevent obesity in men living in Karachi?
• Problems:
– Study design?
– Lifestyle choice outcome is vague
Aims and objectives:
General information
• Begin by summarizing the need for this
research
• Clearly state your main research
objective in the form of aims or a
hypothesis
• Briefly describe how you are going to do
the project in objectives or a paragraph
• Length: 1-2 pages
Aims and objectives:
General information
• After reading your aims section, the
reader should be able to:
– State the need for your project
– Agree that the project is needed
– Summarize the main purpose and broad
methods of your project
– Be excited and interested to read the rest of
your proposal
Questions and discussion
2. Background
Background
• Summarize pertinent literature
– The problem your project is addressing
– Other studies addressing this problem
– Highlight what is missing in the literature
– The research that you are translating
– Support for the theories and methods that
you are using
– End with a clear summary
• Preliminary research
• Length: 1-2 pages
Background
• Your background sections should be:
– Thorough but succinct
– Well-written
– Easy to read and follow
• Subheadings
– Pertinent
Questions and discussion
3. Research Plan
Research plan components
a. Study design
f. Intervention
b. Outcomes
g. Power and
data analysis
plan
c. Study testing
d. Study Sample
e. Timeline
h. Key personnel
i. Ethical Issues
Research plan components
a. Study design
f. Intervention
b. Outcomes
g. Data analysis
plan
c. Study testing
h.
Key
personnel
d. Study Sample
i.
Ethical
Issues
e. Timeline
How to select the best study
design
• Study design
– Should be closely linked to the purpose of
the evaluation
– Should build on a theoretical framework of
behavioral change or other relevant strategy
– Should consider randomized designs first
– Non-randomized designs or designs where the
unit of randomization is other than the
participant may be the most appropriate for
implementation research questions
Study design selection
• Challenge in implementation research is
that the interventions are usually
complex (multifaceted with simultaneous
changes in different parts of the
organization)
• Researcher has variable control over how
the intervention is implemented
Eccles M, et al. Qual Saf Health Care 2003;12;47-52
Critical steps in the research plan
before the definitive study
• Development of a theoretical basis for
the intervention
• Define the components of the
intervention
• Exploratory studies of observational data
+ qualitative research to further refine
the intervention and planned evaluation
• Definitive evaluation
Eccles M, et al. Qual Saf Health Care 2003;12;47-52
Types of studies
• Quantitative non-experimental designs
• Quantitative experimental designs
o Randomization
• Qualitative
• Multi-level, multi-factorial interventions
• Community-based participatory research
Quantitative non-experimental
designs
• Cross-sectional designs
• Uncontrolled before and after
(longitudinal observational cohort
studies)
• Controlled before and after
• Time series analyses
• In translational research there can be
political, practical, and ethical barriers
to randomized designs – but randomized
designs should first be considered
Types of intervention studies
• Randomized or quasi-randomized trials
• Interrupted time series (ITS)
– Defined intervention
– 3 points in time before and after
• Non-randomized studies with controls at a
second site
– Data collected before and after the intervention
– Key to interpretation hinges on comparability of
the sites
• Most are evaluations within systems rather
than between systems
Quantitative experimental designs
• Individual patient-level randomized
controlled trials
• Cluster randomized controlled trials
• Strongest designs to establish a causal
relationship
• Often times difficult to implement…
but considering them is important
Randomized controlled trials
• Considered to be the gold standard
– Randomly allocated to either intervention or
control group
– Best way to insure that both known and unknown
factors that may influence effectiveness of the
intervention are balanced in the two comparison
groups
• Time consuming, expensive, complex, may
require a large number of clusters, tight
inclusion criteria limit generalizabilty
• May not tell you whether an intervention
will improve routine practice
Level of randomization
• Patient
• Health care professional
• Practice/hospital
• Provider group
• Health plan
• Community
LOW LEVEL
HIGH LEVEL
CONTAMINATION
POWER, LOGISTICS
Level of randomization
• At higher levels of randomization measurement of pre-intervention characteristics is
critically important
• Consider stratification on baseline
characteristics that are likely to influence
the effectiveness of the intervention
• Cluster randomization is likely to violate the
assumption of independence of observations
within a cluster
– Two patients in same practice are likely to be more
similar than three from different practices
– Need to be able to estimate the intracluster
correlation coefficient (ICC)
Cluster randomization
• Many quality improvement (QI) interventions
are at provider or system level and if you
randomize at individual patient level it is
likely there will be contamination
• Randomization at higher level will reduce
contamination but you pay a high price with
regard to power and ability to detect
clinically meaningful differences in outcomes;
additionally, risk for bias is much higher
• Randomize at a higher level but collect data
at the patient level
Analysis of cluster randomization
• Analysis at cluster level uses the cluster as
unit of randomization and unit of analysis
– Each cluster is treated as one data point
(inefficient!)
• Patient level analyses that are adjusted for
the cluster
• Patient level analyses that allow for
correlation between clusters is explicitly
modeled; hierarchical nature of data is
accounted for in the analysis
• Unit of randomization must be accounted
for in all analyses
Qualitative studies
• Results from qualitative studies can provide
critical information in the translation of
interventions to real world settings that have
been shown to be effective in clinical trials
• Meanings, definitions, and characteristics of
phenomenon
• Increase acceptability, cultural
appropriateness, and feasibility
• Relying on current literature may not be
sufficient
Types of qualitative research
1. Direct observation
– Observe but do not interact
Types of qualitative research
1. Direct observation
2. Written documents
Types of qualitative research
1. Direct observation
2. Written documents
3. In-depth interviews
– Individual interviews
– Focus group discussions
Questions to ask before including
qualitative research in your study
• Is qualitative research the best way to
answer your research question?
• Is the methodology appropriate to the
research question?
– Justify your research method
Mixed Methods Studies
• “Quantitative research excels at
summarizing large amounts of data and
reaching generalizations based on
statistical perceptions. Qualitative
research excels at “telling the story”
from the participant’s viewpoint,
providing the rich descriptive detail that
sets quantitative results into their human
context.”
Trochim, W.M.K., Research Methods Knowledge Base: www.socialresearchmethods.net
Background premises for multilevel, multi-factorial interventions
• We live in contexts: patients, families,
providers, clinics, health systems,
societies
• Behavior change and quality
improvement are difficult
• Multiple interventions are more
effective than single
• Diabetes and obesity are chronic
conditions: chronic care / chronic
disease models
Community-based participatory
research
• Community focus: often vulnerable or
hard to reach populations
• Collaboration: community and academic
partners
• Equal relationships
• Benefit of community
– Ultimately interventions
– Reduce disparities
Research plan components
a. Study design
f. Intervention
b. Outcomes
g. Data analysis
plan
c. Study testing
h.
Key
personnel
d. Study Sample
i.
Ethical
Issues
e. Timeline
Study outcomes
• One primary outcome
• One or more secondary outcomes
• Short-term, intermediate, and long-term
outcomes
Aim: To evaluate the effectiveness, costeffectiveness, and sustainability of a communitybased diabetes prevention program in Chennai,
India using a randomized controlled trial, with
700 people with pre-diabetes randomized to
either standard lifestyle advice or a culturally
specific, low-cost, intensive lifestyle intervention
– Objective 1: To evaluate effectiveness of the
lifestyle intervention by assessing between
group changes in the following:
• Primary Outcome: incidence of T2DM
• Secondary Outcomes: body weight, percent
body fat, body mass index, waist-to-hip
ratio, fasting glucose, blood pressure,
plasma lipids, activity, and diet
Health care outcomes continuum
Immediate
Outcomes
Learning
Knowledge
Skill Acquisition
Intermediate
Outcomes
Post-Intermediate
Outcomes
Behavior
Change
Improved
Clinical
Indicators
Long Term
Outcomes
Improved
Health
Status
Study outcomes
• Can be quantitative or qualitative in
nature
• Should be sensible based on study design
and research question
• Should be practical
What is practical?
• Reliable and valid
• Succinct and/or engaging
• Relevant to setting and question
• Broadly applicable: literacy, culture,
language
• Reasonable burden on the study
participants
Complexity: The magic diet pill
Dissemination Step
Concept
% Impacted
50% of clinics use
Adoption
50%
50% of clinicians prescribe
Adoption
25%
50% of patients accept
medication
Reach
12.5%
50% follow regimen correctly
Implementation
6.2%
50% of those taking
correctly benefit
Effectiveness
3.2%
50% continue to benefit
after 6 months
Maintenance
1.6%
Re-aim dimensions and
definitions
Individual Level
DIMENSION
REACH
EFFICACY /
EFFECTIVENESS
www.re-aim.org
DEFINITION
• Participation rate among eligible
individuals
• Representativeness of
participants
• Effects on primary outcome of
interest
• Impact on quality of life and
negative outcomes
Re-aim dimensions and
definitions
Both
Setting Level
DIMENSION
ADOPTION
IMPLEMENTATION
MAINTENANCE
www.re-aim.org
DEFINITION
• Participation rate among possible
settings
• Representativeness of settings
participating
• Extent to which intervention
delivered as intended
• Time and costs of intervention
• (Individual) Long-term effects of
intervention ( 6 months
• (Individual) Impact of attrition on
outcomes
• (Setting) Extent of continuation or
modification of treatment
Why multiple measures?
• Clinicians and policy makers focus on
different issues than much efficacy
• Research
–
–
–
–
Biological vs. economic outcomes
Short vs. long term
Patient centered vs. provider/system
Patient level vs. setting level
Examples of outcomes
• Quantitative Measures
• Qualitative Measures
• Types of Outcomes
–
–
–
–
Biological
Psychosocial
Behavioral
Economic
Qualitative measures
• Discussion themes
• Opinions
• Knowledge
• Program acceptability
Biological outcomes
• Examples
–
–
–
–
–
Diabetes incidence
Blood glucose
Hemoglobin A1c
Weight
Blood pressure
Psychosocial outcomes
• Examples
–
–
–
–
Quality of life
Community integration
Depression
Anxiety
Behavioral outcomes
• Examples
– Adherence
– Stage of change
– Behavior change
Economic Outcomes
• Examples
– Cost
– Cost-effectiveness
– Cost-utility
Research plan components
a. Study design
f. Intervention
b. Outcomes
g. Data analysis
plan
c. Study testing
h.
Key
personnel
d. Study Sample
i.
Ethical
Issues
e. Timeline
Measuring study outcomes
• You should plan for the following when
designing your study:
–
–
–
–
How the outcomes will be measured
Use validated tests/study instruments
Timing of study tests
Be conservative about the amount of change
What to include in your grant
• Table of all study tests and test schedule
• Describe how you will measure all
primary and secondary outcomes, as well
as other outcomes
• Include descriptions of screening,
baseline, follow-up testing
• Test for randomization, adherence,
recruitment
Study testing
• Clearly state how you will collect data
What methods will be used to record data?
Who will collect the data?
When and where will data collection occur?
Be systematic in your data collection and
reporting
– Explain how your choices relate to your
research question
–
–
–
–
Research plan components
a. Study design
f. Intervention
b. Outcomes
g. Data analysis
plan
c. Study testing
h.
Key
personnel
d. Study Sample
i.
Ethical
Issues
e. Timeline
Study sample
• Your sample should reflect the larger target
community
• Exclusion and inclusion criteria
• Recruitment
– Who is your study population?
– How and where will you reach them?
– In qualitative research want to recruit an
information-rich sample
Sample size
• Power - probability of rejecting a false
null hypothesis (1-)
• Depends on study design and method of
randomization, desired power, type I
error rate, and the outcome being
studied
– Software programs: SIZ, PASS, Egret
– Online sample size calculators
Sample size
• Examples of online sample size
calculators (see handout):
– http://www.openepi.com/Menu/OpenEpiMenu.htm
– http://stat.ubc.ca/~rollin/stats/ssize/
Research plan components
a. Study design
f. Intervention
b. Outcomes
g. Data analysis
plan
c. Study testing
h.
Key
personnel
d. Study Sample
i.
Ethical
Issues
e. Timeline
Study timeline
Months
Activity
Recruitment and
screening
Baseline testing and
randomization
1
2
3
4
5
6
7
8
9
Research plan components
a. Study design
f. Intervention
b. Outcomes
g. Data analysis
plan
c. Study testing
h.
Key
personnel
d. Study Sample
i.
Ethical
Issues
e. Timeline
Intervention
• Describe intervention and control arm
– Do not re-invent the wheel - use tested
interventions when possible
• Randomization procedures
• Theoretical framework
Research plan components
a. Study design
f. Intervention
b. Outcomes
g. Data analysis
plan
c. Study testing
h.
Key
personnel
d. Study Sample
i.
Ethical
Issues
e. Timeline
Data analysis and data
management software
• Data entry and management:
– Excel, Access
• Textual data analysis:
– Nudist, Atlas/ti, MAXqda
• Quantitative data analysis
– Epi Info, SAS, SUDAAN, Stata, WesVar, SPSS
Data analysis plan
• A data analysis plan should include
descriptions of:
– How and in what units outcomes will be
measured
– How randomization, response rate,
retention, and adherence will be measured
– How data quality will be assured and
assessed
– How data will be protected
Data analysis plan
• A data analysis plan should include
descriptions of (cont.):
– How data will be analyzed
– Quantitative data: treatment of
continuous and categorical variable,
planned statistical tests
– Qualitative data: and/or textual data
analysis description
– List any data analysis or management
programs that will be used
Data analysis plan
• Be specific about and reference
procedures for data analysis
• For qualitative research
– Be clear about how themes are derived
– Be systematic in your analysis to avoid bias
• Consult an expert
Research plan components
a. Study design
f. Intervention
b. Outcomes
g. Data analysis
plan
c. Study testing
h.
Key
personnel
d. Study Sample
i.
Ethical
Issues
e. Timeline
Key personnel
• Include a qualified and well-rounded
study team
• Role in project
• Relevant research
Research plan components
a. Study design
f. Intervention
b. Outcomes
g. Data analysis
plan
c. Study testing
h.
Key
personnel
d. Study Sample
i.
Ethical
Issues
e. Timeline
Ethical Issues
• Subject population
• Sources of research materials
• Data and safety monitoring plan
• Potential risks
• Anticipated benefits
Human subjects
• Procedures for protection of human
subjects
• Procedures for ensuring confidentiality
• Consent procedures
• Study impact
Resources
• http://www.hhs.gov/ohrp/
– Informed consent checklist:
http://www.hhs.gov/ohrp/humansubjects/as
surance/consentckls.htm
• Institutional Review Boards/Research
Oversight Committees
Questions and discussion
4. Sustainability
Sustainability
Translation research is:
“comprehensive, applied research that strives
to translate the available knowledge and
make it useful in everyday clinical and public
health practice”
Narayan KM, et al. Ann Int Med 2004; 140: 958-963.
Sustainability
• Should include:
– Sustainability should be integrated in the
study
– Detailed description of how the program will
be sustained
• What is sustainable?
• Who will continue the program after the
research is complete - community members,
the research institution, etc.
• Is the project scalable?
– Future impact
Questions and discussion
5. Budget
Budget
• Start early
• Personnel, travel, equipment, testing
Budget justification
• Justify the budget items
• Explain how costs were derived
– If the study team is international, how is
salary being determined at each study site
• Explain why these items are needed
• List percent effort
Questions and discussion
Overarching Issues
Grant writing: overarching issues
• Anticipate problems and put plans in
place to avoid them
• Minimize variation
• List limitations and how they will be
minimized
• Lay out study costs and milestones
clearly
Common errors in IDF
applications: pitfalls to avoid
1. Not in the RFA’s framework
2. Not enough detail
3. Lack of cohesion
4. Readability
2. Not enough detail
•
“no clear view of reach…who will participate and why”
•
“unclear which scales will be used”
•
“no mention how data will be obtained”
•
“lack of clear definitions” for outcome measures
•
No description of measurement tools
•
“proposal lacks significant details”
•
No power calculations
•
No support for the feasibility of the study tests and/or
intervention
•
“theoretical framework…is not defined”
3. Disagreement between
different parts of the grant
• “lack of cohesion between” objectives,
study measures, and outcomes
• “timeline and figure” are “somewhat
inconsistent” with the study design
• Costs do not reflect all components of
the study
4. Readability
• “hypotheses are not well articulated”
• “cursory nature of the written proposal”
• Inconsistency in terminology
Avoiding common pitfalls
• Read the grant announcement carefully
• Proof-read several times
• Recruit other readers
Questions and discussion
Checklists for Research Proposals
• Translational Research Checklist
• STROBE
• QUOROM
Grant proposal development:
Final thoughts
•
•
•
•
Start early and develop a timeline
Think of the specific question(s) clearly
Do a good literature search on the topic
Write a one-page summary with
objectives and get feedback from
colleagues
• Meet and discuss study plan
• Write and rewrite 1-2 page purpose and
objectives
Grant proposal development:
Final thoughts
• Clearly draft the plan of investigation
• Get it reviewed and criticized
• Redraft plan of investigation
• Write the background
• Use tables for measurements
• Use diagrams for concept and study
design
Grants’ proposal development:
Final thoughts
• Write, read, and revise several times
• Get different people to read over and
comment
• Check spelling, grammar
• Don’t repeat
• Write short (but thoroughly)!
• Use summaries and highlight key points
• Make it interesting to the reviewer
• On the last two days, be compulsive
Questions?