Progress on use of MAST
in Renewing Health
Kristian Kidholm
Odense University Hospital,
Denmark
Content
1.
Introduction to MAST
2.
Empirical test of MAST in Renewing Health
3.
Problems faced and solutions:
1.
2.
3.
4.
5.
4.
How to ensure that studies are well designed?
Agreement on use of outcome measures?
How to ensure data quality?
Agreement on reporting of results?
The overall result – how to identify European added value?
Use of MAST in other projects
MAST – definition of assessment
If the purposes of an assessment of telemedicine applications are:
– To describe effectiveness and contribution to quality of care
AND
– To produce a basis for decision making
Then the relevant assessment is:
A multidisciplinary process that summarizes and evaluates information
about the medical, social, economic and ethical issues related to the
use of telemedicine in a systematic, unbiased, robust manner.
3
MAST – definition of assessment
If the purposes of an assessment of telemedicine applications are:
– To describe effectiveness and contribution to quality of care
AND
– To produce a basis for decision making
Based on HTA
(EUnetHTA)
Then the relevant assessment is:
A multidisciplinary process that summarizes and evaluates information
about the medical, social, economic and ethical issues related to the
use of telemedicine in a systematic, unbiased, robust manner.
Based on scientific methods and studies
4
Elements in MAST
STEP 1:
Preceding assessment:
• Are you the right one to do the assessment right now?
• Eg. Legal issues, reimbursement, maturity, number of patients
STEP 2:
STEP 3:
5
Multidisciplinary assessment (domains):
Transferability
assessment:
1. Health problem and characteristics of the application
• Cross-border
2. Safety
3. Clinical effectiveness
4. Patient perspectives
5. Economic aspects
6. Organisational aspects
7. Socio-cultural, ethical and legal aspects
• Scalability
• Generalizability
Step 2: Design and methods for data collection
The design should reflect the aim: Estimate effectiveness and contribution to quality
Use highest possible level of evidence (Davies and Newman, 2011)
Safety, clinical, economic and patient outcomes:
- Cluster RCT
6
- Pragmatic RCT
If RCT is not feasible: Use other designs:
– Quasi-experimental
comparison of different
hospitals/units
– Uncontrolled
“before and after” studies
Risk of systematic differences
Follow guidelines for data
collection, analysis and reporting
Strengths and weakness
Weaknesses of MAST:
• Time consuming
• Focused on outcomes
• Only relevant in assessment of matured telemedicine applications.
Strengths of MAST:
• Based on the requests and comments from stakeholders
• Multidisciplinary and comprehensive
• Based on scientific studies and criteria for quality
• Based on HTA (EUnetHTA): Familiar to stakeholders in EU, hospitals..
The result of MAST for decision makers?
1.
Problem,
Application
2.
Safety
3.
Clinical
4.
Patient
5.
Economic
6.
Organizational
7.
Sociocultural
Describe
Evidence?
Outcome?
Evidence?
Outcome?
Evidence?
Outcome?
Evidence?
Outcome?
Evidence?
Outcome?
Evidence?
Outcome?
1 2 3 4 5 6 7
1 2 3 4 5 6 7
1 2 3 4 5 6 7
Empirical test of MAST in
Renewing Health
EC project: Renewing Health
Objective:
– Large scale, real life implementation of telemedicine services
– Assessment of outcomes based on MAST
Budget: 14 mill EURO
Pilots:
– 20 pilots in 9 European regions (I, DK, S, N, ES, GR, D, A, FIN)
– Patients: 7.900 patients with COPD, diabetes, CHF
Example: Assessment of the
COPD suitcase in Denmark
•
•
•
•
Safety
Clinical effectiveness
Mortality
FEV1, SAT, MRC, BMI
SF-36
Exercise
Patient perspectives
WSD acceptability questionnaire
Qualitative interviews
Economic aspects
Investments
Number of consultations
Number of telephone calls
Number of readmissions, bed days
Number of outpatient visits
Number of home nurse visits
Use of emergency ward
Changes in revenue (DRG)
• Organisational aspects
Interview with nurses on task shifts,
use of time, satisfaction etc.
Transferability assessment: Comparison of DK, E, GR
DESIGN: RCT,
similar for
intervention
and control group
(n = 266)
Individual pilots in Renewing Health
Diabetes patients
Cluster 1: Medium term monitoring
S
N
Cluster 2: Long term monitoring
D
GR
Cluster 3: Ulcer monitoring
DK
SF
COPD patients
Cluster 4: Short term, after discharge
GR
ES
DK
Cluster 5: Long term monitoring
I
A
D
Cluster 6: Medium term monitoring
S
SF
Cluster 7: Remote monitoring CHF
I
GR
Cluster 8: Remote monitoring ICD
I
(DK)
Cluster 9: Remote monitoring ACT
I
Heart patients
A
Problems and solutions
1. How to ensure that studies are well designed?
•
•
Almost all studies are pragmatic RCT’s
Common scientific protocols (CONSORT, MAST):
1. Oobjectives and the trial type
2. Planned sample sizes
3. Trial start and end dates
4. Eligibility criteria
5. Enrolment modalities
6. Description of the randomisation methodology
7. Demographic and clinical baseline characteristics
8. Interventions
9. Primary and secondary outcomes
10.Evaluation time points
11.Economical evaluation
12.Evaluation of perception of health care professionals
13.Evaluation of patient satisfaction
14.Additional evaluations
15.Statistical analysis
Problems and solutions
2. Agreement on use of outcome measures?
•
•
Similar primary outcomes within each cluster
Minimum dataset in all studies
Demographic data:
Based on WHO project STEPS
Clinical effectiveness:
Health related quality of life – SF36
Patient perception:
WSD patient acceptability questionnaire
Problems and solutions
2. Agreement on use of outcome measures?
•
•
Similar primary outcomes within each cluster
Minimum dataset in all studies
Economic aspects:
• Investments in the telemedicine application
• Running costs of delivering telemedicine and comparator:
• Each patient's use of health care service:
Number of admissions
Number of bed days
Number of GP visits
Number of visits to emergency department
•Reimbursement of the telemedicine service (business case)
Problems and solutions
2. Agreement on use of outcome measures?
•
•
Similar primary outcomes within each cluster
Minimum dataset in all studies
Organizational aspects:
•
Effects on work processes:
– Workflow and task shifting
– Training
– Communication
•
Effects on structural outcomes:
– Description/number of units collaborating
– Changes in organisation
– Changes in geographical spread
•
Cultural outcomes:
– Staff attitudes towards the application
Common list of questions
Problems and solutions
3. How to ensure data quality?
1.
2.
3.
4.
Similar coding of common variables
Collection of CRF from all pilots
Assistance in development of Epidata Databases
Monthly monitoring of data collection in each pilot
•
•
•
Based on GCP
Number of patients
Completeness
S1
2500
N1
SF1
2000
Patients
A1
1500
D2
GR2
1000
DK3
GR4
500
ES4
0
DK4
0
1
2
3
4
5
6
Month
7
8
9
10
11
12
I5
A5
Problems and solutions
3. How to ensure data quality?
700
600
GR2
DK3
Patients
500
GR4
400
DK4
300
GR7
S1
200
S6
N1
100
0
0
2
4
6
8
Months
10
12
14
Problems and solutions
4. Agreement on reporting of results?
Guide for analysis of results within each domain:
1. Health problem and characteristics of the application
2. Safety (adverse effects)
Based on STARE-HI + CONSORT
3. Clinical effectiveness
4. Patient perspectives
Based on validation studies by WSD
5. Economic aspects
Based on guide by Drummond et al 2005
6. Organisational aspects
Based on guide for organizational studies
7. Socio-cultural, ethical and legal aspects
Problems and solutions
5. Comparison of results – how to identify European added value?
Meta-analysis of results from pilots within a cluster
Study
Outcome
RH
No 1
RH
No 2
RH
No 3
Other
No 4
No. Of patients
100
200
600
32
Clinical outcome
- 2%
+ 1%
+5 %*
-1%
SF-36 dimension X (0-100)
+5
+ 10 *
+8*
-2
Patient acceptability (0-100)
-10
+6
+ 8*
Readmissions
0,1
-1,2
- 2,3*
0,3
Costs per patient (€)
2200
1500
850
2300
Interpretation:
- Some show positive effect, others negative
- Some statistically significant, others not
- Some studies are large, others not
Problems and solutions
5. Comparison of results – how to identify European added value?
Meta-analysis of results from pilots within a cluster
•
A statistical analysis of results from independent studies,
which aims to produce a single estimate of a treatment effect
•
Account can be taken of heterogeneity of the studies:
•
Inclusion of characteristics of patients (meta level or individual patient-data)
Problems and solutions
5. Comparison of results – how to identify European added value?
Meta-analysis of results from pilots within a cluster
Reasons for meta-analysis:
•
Obtain more precise estimate of the effect of an intervention
•
Increase the statistical power (ability to find stat. significant effects)
•
Increase generelisability
•
Increase possibilities for subgroup analysis
•
Used by Cochrane Collaboration, NICE etc.
Use of MAST in other projects
• Jose Asua Batarrita, INAHTA Board Director, at WoHIT 2011.
•
Approach used in “...a number of telemedicine projects related to home
telecare of frail elderly patients with heart failure and COPD,
teledermatology, teleoncology and teleophthalmology “
Use of MAST in other projects
•
MAST is recommended as the assessment model
in the national telemedicine strategy by
the Association of Danish Regions
•
Renewing Health is also collaborating with inCASA:
Questions?
[email protected]
www.renewinghealth.eu