Dashboard Review
First Quarter of FY 2015
Joe Selby, MD, MPH
Executive Director
Michele Orza, ScD
Senior Advisor to the Executive Director
Presentation Overview
• Evolution of our Board of Governors’ Dashboard from 2013
• Latest revisions
• Overall organization and types of graphics
• Noteworthy and Yellow-flagged items
• Examples of background slides
• Examples of metrics proposed for Committee dashboards
Discussion Questions
• Are there metrics that should be revised or retired or
moved to the Committee Dashboards?
• Are there new or different metrics suggested by today’s
presentation and discussion? If so, are they appropriate for
the Board or one of the Committees?
• What further improvements should we consider so that our
Dashboard and accompanying materials effectively convey
the status of our work and progress toward our goals?
Our End-of-CY 2013 Dashboard
Our End-of-FY 2014 Dashboard
*Influencing Research*
PCORI inspired the American Heart Association to pilot the Citizen Scientist Think Tank,
intended to foster patient engagement in the design of heart disease and stroke research
Board of Governors
FY2015 Dashboard – Q1
(As of 12/31/2014)
Our Goals: Increase Information, Speed Implementation, and Influence Research
Funds Committed to Research – Budget=$640M
Q1
Q2
Q3
Q4
Broad
100
90
80
70
60
50
40
30
20
10
0
Number of Projects
Budget
20
Actual
10
0
100
200
300
400
$ Millions
500
600
0
Q1
Q2
Completion of Projects
Q3
NA=Not Applicable
Q4=Q4 2014
Q1=Q1 2015
Q2
Q3
Q4
Q1
NA
Research Engagement
Q4
PPRNs
CDRNs
Uptake of Methodology Standards
Journal Articles Published
30
35
Q2
7,000
30
Expected
Q2
30
Actual
Q3
6,000
25
Q3
Q4
5,000
25
Number of Articles
Number of Projects
NA
Needs Attention
Q2=Q2 2014
Q3=Q3 2014
Percent Meeting All
30
Pragmatic
Engagement
Off Target
Percent of Projects Meeting All Milestones
Projects Awarded
Targeted
PCORnet
Legend
On Target
20
15
10
5
0
0
20
Q1
Q1
Q2
Q3
15
3,000
10
Q2
5
1,000
0
By Awardees
Q4
Q3
10
2,000
0
Web Views
About or By PCORI
Citations
Progress of PCORnet
Expenditures – Total Budget=$362M
Q1
Q1
4,000
0
0
Q4
20
Q4
Research
Budget
Version
2.0 of
CDM
Complete
Phase II
PFA
Released
Research
Actual
Other
Budget
Obesity
Cohort
Project
Awarded
Phase II
awarded
Q1
Other
Actual
=Target
0
50
100
150
$ Millions
200
250
=Actual
Q1 Governance
policies
approved
Q2
Aspirin
Trial
Awarded
Q3
Health
Systems
Projects
Awarded
Concerns with FY 2014 Dashboard
and Revisions in FY 2015 Dashboard
Concern with FY 2014 Dashboard
Revision in FY 2015 Dashboard
Cluttered – too many items
Cut from 17 boxes to 9
Hard to read
Remaining items enlarged
Arrow graphics difficult to interpret
Replaced with horizontal bar charts
PCORnet arrow provided no information
PCORnet timeline with key milestones
Many metrics not right level for Board
• Focused more on key Outputs and Goals
• Others moved to Committee Dashboards
*Influencing Research*
PCORI inspired the American Heart Association to pilot the Citizen Scientist Think Tank,
intended to foster patient engagement in the design of heart disease and stroke research
Board of Governors
FY2015 Dashboard – Q1
(As of 12/31/2014)
Our Goals: Increase Information, Speed Implementation, and Influence Research
Funds Committed to Research – Budget=$640M
Q1
Q2
Q3
Q4
Broad
100
90
80
70
60
50
40
30
20
10
0
Number of Projects
Budget
20
Actual
10
0
100
200
300
400
$ Millions
500
600
0
Q1
Q2
Completion of Projects
Q3
NA=Not Applicable
Q4=Q4 2014
Q1=Q1 2015
Q2
Q3
Q4
Q1
NA
Research Engagement
Q4
PPRNs
CDRNs
Uptake of Methodology Standards
Journal Articles Published
30
35
Q2
7,000
30
Expected
Q2
30
Actual
Q3
6,000
25
Q3
Q4
5,000
25
Number of Articles
Number of Projects
NA
Needs Attention
Q2=Q2 2014
Q3=Q3 2014
Percent Meeting All
30
Pragmatic
Engagement
Off Target
Percent of Projects Meeting All Milestones
Projects Awarded
Targeted
PCORnet
Legend
On Target
20
15
10
5
0
0
20
Q1
Q1
Q2
Q3
15
3,000
10
Q2
5
1,000
0
By Awardees
Q4
Q3
10
2,000
0
Web Views
About or By PCORI
Citations
Progress of PCORnet
Expenditures – Total Budget=$362M
Q1
Q1
4,000
0
0
Q4
20
Q4
Research
Budget
Version
2.0 of
CDM
Complete
Phase II
PFA
Released
Research
Actual
Other
Budget
Obesity
Cohort
Project
Awarded
Phase II
awarded
Q1
Other
Actual
=Target
0
50
100
150
$ Millions
200
250
=Actual
Q1 Governance
policies
approved
Q2
Aspirin
Trial
Awarded
Q3
Health
Systems
Projects
Awarded
*Influencing Research*
PCORI inspired the American Heart Association to pilot the Citizen Scientist Think Tank,
intended to foster patient engagement in the design of heart disease and stroke research
Board of Governors
FY2015 Dashboard – Q1
(As of 12/31/2014)
Our Goals: Increase Information, Speed Implementation, and Influence Research
Funds Committed to Research – Budget=$640M
Q1
Q2
Q3
Q4
Broad
100
90
80
70
60
50
40
30
20
10
0
Number of Projects
Budget
20
Actual
10
0
100
200
300
400
$ Millions
500
600
0
Q1
Q2
Completion of Projects
Q3
NA=Not Applicable
Q4=Q4 2014
Q1=Q1 2015
Q2
Q3
Q4
Q1
NA
Research Engagement
Q4
PPRNs
CDRNs
Uptake of Methodology Standards
Journal Articles Published
30
35
Q2
7,000
30
Expected
Q2
30
Actual
Q3
6,000
25
Q3
Q4
5,000
25
Number of Articles
Number of Projects
NA
Needs Attention
Q2=Q2 2014
Q3=Q3 2014
Percent Meeting All
30
Pragmatic
Engagement
Off Target
Percent of Projects Meeting All Milestones
Projects Awarded
Targeted
PCORnet
Legend
On Target
20
15
10
5
0
0
20
Q1
Q1
Q2
Q3
15
3,000
10
Q2
5
1,000
0
By Awardees
Q4
Q3
10
2,000
0
Web Views
About or By PCORI
Citations
Progress of PCORnet
Expenditures – Total Budget=$362M
Q1
Q1
4,000
0
0
Q4
20
Q4
Research
Budget
Version
2.0 of
CDM
Complete
Phase II
PFA
Released
Research
Actual
Other
Budget
Obesity
Cohort
Project
Awarded
Phase II
awarded
Q1
Other
Actual
=Target
0
50
100
150
$ Millions
200
250
=Actual
Q1 Governance
policies
approved
Q2
Aspirin
Trial
Awarded
Q3
Health
Systems
Projects
Awarded
*Influencing Research*
PCORI inspired the American Heart Association to pilot the Citizen Scientist Think Tank,
intended to foster patient engagement in the design of heart disease and stroke research
Board of Governors
FY2015 Dashboard – Q1
(As of 12/31/2014)
Our Goals: Increase Information, Speed Implementation, and Influence Research
Funds Committed to Research – Budget=$640M
Q1
Q2
Q3
Q4
Broad
100
90
80
70
60
50
40
30
20
10
0
Number of Projects
Budget
20
Actual
10
0
100
200
300
400
$ Millions
500
600
0
Q1
Q2
Completion of Projects
Q3
NA=Not Applicable
Q4=Q4 2014
Q1=Q1 2015
Q2
Q3
Q4
Q1
NA
Research Engagement
Q4
PPRNs
CDRNs
Uptake of Methodology Standards
Journal Articles Published
30
35
Q2
7,000
30
Expected
Q2
30
Actual
Q3
6,000
25
Q3
Q4
5,000
25
Number of Articles
Number of Projects
NA
Needs Attention
Q2=Q2 2014
Q3=Q3 2014
Percent Meeting All
30
Pragmatic
Engagement
Off Target
Percent of Projects Meeting All Milestones
Projects Awarded
Targeted
PCORnet
Legend
On Target
20
15
10
5
0
0
20
Q1
Q1
Q2
Q3
15
3,000
10
Q2
5
1,000
0
By Awardees
Q4
Q3
10
2,000
0
Web Views
About or By PCORI
Citations
Progress of PCORnet
Expenditures – Total Budget=$362M
Q1
Q1
4,000
0
0
Q4
20
Q4
Research
Budget
Version
2.0 of
CDM
Complete
Phase II
PFA
Released
Research
Actual
Other
Budget
Obesity
Cohort
Project
Awarded
Phase II
awarded
Q1
Other
Actual
=Target
0
50
100
150
$ Millions
200
250
=Actual
Q1 Governance
policies
approved
Q2
Aspirin
Trial
Awarded
Q3
Health
Systems
Projects
Awarded
Early Signs of
Influence on Research
*Influencing Research*
PCORI inspired the American Heart Association to pilot the Citizen Scientist Think Tank,
intended to foster patient engagement in the design of heart disease and stroke research
“As PCORI has evolved, the notion of patient engagement has changed dramatically, and
the American Heart Association (AHA) has sought more ways we can make the patient
the focal point.”*
The AHA was an early adopter of PCORI’s efforts to get people and organizations to focus
on patient-centeredness. For example, they developed the
Citizen Scientist Think Tank (CSTT) Pilot
•
A Community of patient and caregiver volunteers from AHA with availability and
training to help design research studies for heart disease and stroke
•
The pilot involved 30 volunteers, was successful, and AHA’s goal now is to obtain
funding and scale up to 300+
•
The AHA explained that, although they are a patient advocacy organization, most of
the research they fund is designed by clinicians and researchers, and the CSTT is
intended to ensure that patients have a voice and the appropriate resources to
engage in the research process
*Marko Mijic, Government Relations Manager, American Heart Association
Funds Committed
by Type of Project and Fiscal Year
350
Q1
Q2
Q3
Q4
Budget
Actual
0
100
200
300
400
$ Millions
58
300
Pragmatic PFAs
50
Targeted PFAs
250
Broad PFAs
$Millions
PCORnet
200
196
Pilot Projects
150
244
100
50
96
31
9
FY 2012
FY 2013
0
FY 2014
25
1
FY 2015 Q1
500
600
Q1
Funds Committed by Type of Project:
Cumulative by Fiscal Year
Q2
Q3
Q4
Budget
Actual
0
100
200
300
400
500
600
$ Millions
Pilots
As of Q1 FY 2015
Total=$710M
End of FY 2014
Total=$684
End of FY 2013
Total=$334M
Targeted
Pragmatic
$133M
$108M
$50M
PCORnet
$31M
$31M
$9M
Broad
$31M
$105M
$244M
$106M
$440M
$440M
30
150
Broad
20
10
0
147
Q1
Broad
PCORnet
Targeted
Pragmatic
Q2
Q3
Engagement
121
125
Number of Projects
Pragmatic
Engagement
Number of Projects
Number and Types
of Projects Awarded
Targeted
PCORnet
100
75
50
29
25
13
7
3
0
2013
2014
Q1 2015
Q2 2015
Q3 2015
Q4 2015
Q4
Targeted
PCORnet
Pragmatic
Engagement
Broad
Number of Projects
30
Engagement Awards
20
10
0
Q1
Q2
Q3
Q4
40
36
35
Q4 FY 2014
Q1 FY 2015
32
30
Number
25
22
20
18
15
10
7
5
3
0
LOIs Received*
Applications Received
Awards Made**
*The January 2015 submission cycle closed on 1/2/15. For this cycle, we received 36 LOIs by the close of Q1 FY
2015 (12/31/14), and an additional 28 LOIs by the close of the cycle (1/2/15).
**Awards include only contracts that were fully executed during the quarter
Percent Meeting All
Progress of Research Projects:
Milestones
100
90
80
70
60
50
40
30
20
10
0
Q2
Q3
Q4
Q1
NA
Research Engagement
2014
PPRNs
CDRNs
2015
Q2
Q3
Q4
Q1
Number of Projects with a Progress Report Due
• 62 with 6-month reports
• 87 with 12-month report
• 46 with 18-month reports
57
140
186
195
Percent of Projects Meeting All Milestones Due
63%
51%
45%
40%
Average Percent of Milestones Due that Were Met
87%
85%
84%
83%
We continue to refine our approach to measuring and reporting on the progress of projects
and will present additional analyses as we have sufficient data for them.
Percent Meeting All
Progress of Research Projects:
Additional Measures
100
90
80
70
60
50
40
30
20
10
0
Q2
Q3
Q4
Q1
NA
Research Engagement
PPRNs
CDRNs
100
91
87
90
2015
2014
80
80
Q2
76
Q3
Q4
Q1
70
63
57 56
Percent
60
51
50
40
45
40
30
20
10
4
NA
0
Meeting All
Milestones
Meeting
Recruitment
Milestones
Obtained IRB
Approval on
Schedule
1
1
Payment Hold for
Programmatic
Reasons
3 4
4
4
Contract
Modification for
Milestones
Terminated
Number of Projects
Actual (2014) and Expected (2015 >>)
Number of Contracts Ending per Quarter
(Includes estimates for some contract modifications in process)
35
Expected
30
Actual
25
20
15
10
5
0
0
0
Q1
Q2
Q3
Q4
60
55
Pilot Expected
50
Number of Projects
Pilot Actual
CER Expected
CER Actual
40
Methods Expected
36
Methods Actual
30
25
21
20
18
21
17
12
10
14
13
14
11
8
33
4
4
11
4
2
4
2
0
2014 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
2015
2015
2016
2016
2017
2017
2018
2018
2019
2019
2020
2020
Fiscal Year
30
Number of Journal Articles
Concerning PCORI
Q2
Number of Articles
Q3
Q4
20
Q1
10
0
By Awardees
120
116
110
Q2
100
Q3
Q4
Q1
Number of Articles
90
80
70
60
56
50
40
30
22
23
21
20
10
5
11
8
3
12
5
3
0
Articles Resulting from
PCORI-funded Projects
Articles by or about PCORI
Articles that Cite or Mention
PCORI Work
About or By PCORI
Uptake of Methodology Standards
Q2 2014
30
7,000
6139
26 26
6,000
25
Q3 2014
24
Q4 2014
5,000
4768
4603
Q1 2015
20
4,000
14
15
3009
We are also tracking:
3,000
•
•
•
•
•
10
2,000
5
1,000
0
0
Web Views
Citations
Adoption
Endorsements
CE/CME (Q3)
Uptake into Curriculum
Use of PCORI-developed
Curriculum (2016)
Communications: Website, E-mail, Social
Media, and Media Coverage Metrics
Communications
Website
E-mail
Our Q4 unique website visitors numbers continue to beat our target.
We continue to exceed industry standards for open and click-through
rates by a wide margin.
Monthly Unique Traffic
Website Unique Traffic
Target
60000
Actual
60,000
Email Click-Thru Rate
Email Open Rate
50,000
30%
20%
40,000
40000
10%
20,000
20000
15%
20%
30,000
10%
5%
10,000
Q2
Q3
Q4
Q1
2014 2014 2014 2015
0%
0%
0
0
Q2
2014
2012
2013
Q3
2014
2014
Social Media
Because our early Twitter stats were low, year-to-year comparisons
against targets aren’t useful. We’re tracking follower growth and
impressions and working on a more sophisticated reach analysis.
Q1
2015
Target
Q2
2014
4,000
3,000
2,000
1,000
Q4
2014
Actual
We continue to grow the number of mentions of our work in
general/trade media.
2013
2014
Q1
45
150
Q2
48
212
Q3
89
149
Q4
117
166
6,000
5,000
Q3
2014
Media Coverage
Monthly Twitter Follower Growth
7,000
Q4
2014
2015
157
Q1
2015
PCORI Ambassadors Program
Ambassadors Enrolled by Stakeholder Group
40
(Cumulative)
Ambassadors Enrolled by
Geographic Region
(Cumulative)
35
30
25
20
Q2
Q3
Q4
West: 20
Midwest: 42
Q1
Northeast: 49
15
Southwest: 25
10
Southeast: 29
5
0
Cumulative Total
as of Q1 2015 = 165
Operational Excellence
100
93
88
94
93
Q2
96
90
91
90
Q3
Q4
Q1
92
80
Percent of Target
69*
60
40
37
*100% for Targeted (3/3)
and 67% for Broad (29/43)
20
0
NA
Award to Contract
w/in 90 days
Contracts HelpDesk
Science HelpDesk
100% Customer Satisfaction
Discussion Questions
• Are there metrics that should be revised or retired or
moved to the Committee Dashboards?
• Are there new or different metrics suggested by today’s
presentation and discussion? If so, are they appropriate for
the Board or one of the Committees?
• What further improvements should we consider so that our
Dashboard and accompanying materials effectively convey
the status of our work and progress toward our goals?
*Influencing Research*
PCORI inspired the American Heart Association to pilot the Citizen Scientist Think Tank,
intended to foster patient engagement in the design of heart disease and stroke research
Board of Governors
FY2015 Dashboard – Q1
(As of 12/31/2014)
Our Goals: Increase Information, Speed Implementation, and Influence Research
Funds Committed to Research – Budget=$640M
Q1
Q2
Q3
Q4
Broad
100
90
80
70
60
50
40
30
20
10
0
Number of Projects
Budget
20
Actual
10
0
100
200
300
400
$ Millions
500
600
0
Q1
Q2
Completion of Projects
Q3
NA=Not Applicable
Q4=Q4 2014
Q1=Q1 2015
Q2
Q3
Q4
Q1
NA
Research Engagement
Q4
PPRNs
CDRNs
Uptake of Methodology Standards
Journal Articles Published
30
35
Q2
7,000
30
Expected
Q2
30
Actual
Q3
6,000
25
Q3
Q4
5,000
25
Number of Articles
Number of Projects
NA
Needs Attention
Q2=Q2 2014
Q3=Q3 2014
Percent Meeting All
30
Pragmatic
Engagement
Off Target
Percent of Projects Meeting All Milestones
Projects Awarded
Targeted
PCORnet
Legend
On Target
20
15
10
5
0
0
20
Q1
Q1
Q2
Q3
15
3,000
10
Q2
5
1,000
0
By Awardees
Q4
Q3
10
2,000
0
Web Views
About or By PCORI
Citations
Progress of PCORnet
Expenditures – Total Budget=$362M
Q1
Q1
4,000
0
0
Q4
20
Q4
Research
Budget
Version
2.0 of
CDM
Complete
Phase II
PFA
Released
Research
Actual
Other
Budget
Obesity
Cohort
Project
Awarded
Phase II
awarded
Q1
Other
Actual
=Target
0
50
100
150
$ Millions
200
250
=Actual
Q1 Governance
policies
approved
Q2
Aspirin
Trial
Awarded
Q3
Health
Systems
Projects
Awarded