To crawl before we run:
optimising therapies with
aggregated data
Chris Evans, Michael Barkham,
John Mellor-Clark, Frank Margison,
Janice Connell
Aims
Panel aim is to help bridge the gap between
researchers and practitioners
Specifically, to promote new forms of “practice
based evidence” (PBE) which work in and
across that gap and which complement EBP
This paper aims to present low sophistication,
service oriented methods to complement the
HLM and other sophisticated methods that
Wolfgang, Zoran and many others have
developed
Specific aims for this presentation
Show realities of routine data collection
Show the magnitude of service level variation
Argue that simple service level analyses can
help us learn from treatment failures
Computer processing is needed by most
services/practitioners but is alien to many, two
methods of computer processing available for
CORE
For now, confidence intervals and graphical
data presentations may be the “zone of
proximal development”
The dataset
6610 records (from >12k):
33 primary care NHS services
40 to 932 records per service
Anonymised, voluntary
Four components to the data:
Therapist completed CORE-A
Therapy Assessment Form (TAF)
End of Therapy Form (EOT)
Client completed CORE-OM
At assessment and end of therapy or follow-up
CORE-A TAF
CORE-A EOT
CORE-OM
CORE-PC version of CORE-OM
“It’s really simple and easy to use. I’m not very computer literate, but I’d got
to grips with it in less than an hour”
0
20
1496
40
2
60
4
5
1805
387
1
1564
1087
448
7
1040
1362
1389
3008
1385
1823
%
Services
100
203
532
2021
1744
950
1657
1232
1819
968
1308
329
915
1379
1086
812
927
80
Plotting data: simple proportion
% w ith the second CORE-A (EOT) form
0
20
1496
40
2
60
4
5
1805
387
1
1564
1087
448
7
1040
1362
1389
3008
1385
1823
%
Services
100
203
532
2021
1744
950
1657
1232
1819
968
1308
329
915
1379
1086
812
927
80
Plotting data: reference lines
% w ith the second CORE-A (EOT) form
Plotting data: add CI for sites
932
55
51 60
102 62
69
69
40
0
20
315
Services
165
102
203
532
2021
1744
950
1657
1819
1232
1308
968
329
915
1379
1086
5
4
1805
387
690
203101 77
639
98
141
323142
1
1087
1564
448
7
1362
1040
1389
3008
1823
1496
40
128
49
135
164 75
168
113
286
430
153
2
%
60
1385
80
927
812
100
% w ith the second CORE-A (EOT) form
173
300
Plotting data: add summary
932
55
51 60
203
532
2021
1744
950
1657
1819
1232
1308
968
329
915
1379
1086
5
4
1805
387
173
300
690
203101 77
639
98
141
323142
1
1087
1564
448
7
1362
1040
1389
3008
1823
1496
40
128
49
135
164 75
168
113
286
430
153
2
%
60
1385
80
927
812
100
% w ith the second CORE-A (EOT) form
102 62
165
102
69
69
40
Overall proportion = 72.1%
Maximum = 98.8%
Minimum = 33%
Ratio, max:min = 2.99
Chi square = 1181.75 d.f. = 32 p = 0
Number "significantly" high 13
Number "significantly" low 10
Number "significantly" different 23
0
20
315
Services
Getting data (2): CORE-OM 1
60
69 75
98
165
102
1744
448
1362
1379
1086
62
915
203
927
1657
387
812
1819
1496
1564
950
1232
2
1385
2021
1805
329
1040
1389
968
1823
532
1308
7
4
1
315
141
102
173
101
49 55
1087
40
40
%
60
430
286168142
164135
690
932
203
323
639153113
77 69
300128
5
80
3008
100
% w ith CORE-OM at assessment
Overall proportion = 88.3%
Maximum = 100%
Minimum = 39.2%
Ratio, max:min = 2.55
Chi square = 396.43 d.f. = 32 p = 0
Number "significantly" high 13
Number "significantly" low 9
Number "significantly" different 22
0
20
51
Services
Getting data (3): CORE-OM 2
1496
532
2
1379
1744
1308
203
1564
968
1232
915
812
4
329
927
950
1819
98
639
142101
173
1
5
69
690
323
102102
168
203
62 60
Overall proportion = 39.2%
Maximum = 65.5%
Minimum = 9.33%
Ratio, max:min = 7.02
Chi square = 444.33 d.f. = 32 p = 0
Number "significantly" high 7
Number "significantly" low 9
Number "significantly" different 16
153135128 77 49
51 40
315 69
165
7
430164
20
1805
387
1362
2021
1389
3008
1823
1040
1087
448
40
1385
932
141
286
75
113
55
300
0
%
60
1086
1657
80
% w ith final CORE-OM
Services
Getting data (4): all four forms
168
153
77
1657
1379
1744
1308
203
1564
968
915
1232
812
927
950
1819
329
387
69
2
1496
532
69
315
1805
2021
7
4
1040
1389
3008
1
448
62 60
323
203102102
141
75
113
5
1385
1087
40
20
932
98
690
430
286
128
135
164
165
101
639
173142
49
40
55
51
300
0
%
1823
1362
60
1086
% w ith all four forms
Services
Overall proportion = 34.8%
Maximum = 64.2%
Minimum = 8.67%
Ratio, max:min = 7.4
Chi square = 611.05 d.f. = 32 p = 0
Number "significantly" high 13
Number "significantly" low 9
Number "significantly" different 22
Getting data: summary
For each of these basic indices the differences
across services:
were significant p<.0005
were very large in magnitude
the number “significantly” different from overall
proportion ranged from 15 to 22 of the 33
Even at the “best” end, datasets are fairly
incomplete …
… at the “worst” end completion rate is
cripplingly low
Demographics (1): gender
%
1086
1823
1744
1379
1657
927
915
532
812
1308
1496
1819
968
1362
1040
1385
1
5
1087
1389
1564
2
203
950
7
931
60
286
127
51
165
300
315
638
98
102102
142
75
141
40
164153
69
55
0
20
40
69
1232
448
430
690
173203
113
323
168135
77
1805
3008
329
4
387
60
2021
80
100
% Female
Services
49
101 62
Overall proportion = 71.5%
From n = 6607 n(miss) = 3 %(miss) = 0
Higest proportion = 85.5%, lowest = 59.7%
Ratio, max:min = 1.43
Chi square = 63.53 d.f. = 32 p = 0.0008
Number "significantly" high 4
Number "significantly" low 3
Number "significantly" different 7
Demographics (2): ethnicity
1389
1232
224
329
74
62
75
67
75
59
31
49
99
1087
661
40
60
107
303
Overall proportion = 91.1%
From n = 5758 n(miss) = 852 %(miss) = 13
Higest proportion = 100%, lowest = 52.9%
Ratio, max:min = 1.43
Chi square = 686.72 d.f. = 32 p = 0
Number "significantly" high 11
Number "significantly" low 5
Number "significantly" different 16
Dotted red CI indicates %(miss) > 20%
51
Services
448
4
532
968
7
2021
1308
1086
610298
117
411
138
501
169
145179
113
137
117
148
58
127
387
1823
1564
1805
203
950
1040
1819
2
5
1362
1379
1657
101 98
3008
915
1385
812
1496
305
80
%
1
1744
927
100
% White/European referrals
49
Demographics (3): employment
532
3008
5
1086
387
1744
1379
203
2
420
67 56
168134 99
134
598
391304138
111
166
222
92
111
1819
1362
968
812
1232
915
1385
4
1496
1308
329
1657
1805
950
7
664
1
2021
1087
448
1564
1040
1823
1389
60
126
115
301
40
%
927
80
% full-time or part-time employed
96
94
75
48
62
295
72
139
Overall proportion = 55.6%
From n = 5589 n(miss) = 1021 %(miss) = 15
Higest proportion = 74.6%, lowest = 27.3%
Ratio, max:min = 1.43
Chi square = 188.05 d.f. = 32 p = 0
Number "significantly" high 8
Number "significantly" low 9
Number "significantly" different 17
Dotted red CI indicates %(miss) > 20%
49 51
104
20
54
33
Services
Demographics (4): young age
0
1308
1087
1040
1385
812
1232
968
532
1744
915
1379
1805
2
1
3008
7
203
4
950
1819
927
448
1657
1496
141
5
1389
329
5
10
1086
1564
387
15
%
2021
1362
Overall proportion = 4.5%
From n = 6554 n(miss) = 56 %(miss) = 1
Higest proportion = 10.6%, lowest = 0.787%
Ratio, max:min = 1.43
Chi square = 67.48 d.f. = 32 p = 0.0002
Number "significantly" high 1
Number "significantly" low 4
Number "significantly" different 5
1823
20
25
% under 20 years of age
314
62 55
127319 98 164
201150
60
72 141
427284
165102101
113
172134
68
Services
69
40
77 49
633299152
687
925
102
51
Demographics (5): older age
1389
Overall proportion = 6.47%
From n = 6554 n(miss) = 56 %(miss) = 1
Higest proportion = 38.6%, lowest = 0%
Ratio, max:min = 1.43
Chi square = 282.75 d.f. = 32 p = 0
Number "significantly" high 4
Number "significantly" low 3
Number "significantly" different 7
30
40
50
% over 59 years of age
319201172 77 102
51 152 72
150 68 101
1385
1
7
532
968
812
1819
1805
1086
1657
1379
1308
1564
687314164
2
5
1496
1232
927
1087
1040
915
2021
203
950
329
387
448
3008
4
10
0
134 55
1744
20
1823
%
1362
127
60
925
141165 98
Services
633427284
69 49
141
102
113
299
62
40
Demographics (6): age
1389
812
1385
1819
1805
1657
1496
1823
1564
1086
1379
968
7
950
1
3008
1040
1744
1308
532
387
203
329
5
4
1087
1232
127
2
927
2021
40
915
448
45
1362
Overall median = 37 on range from 11 to 89
From n = 6554 n(miss) = 56 %(miss) = 1
Highest median = 46 lowest = 33.5
Ratio, max:min = 1.37
Kruskal-Wallis chi square = 110.2 d.f. = 32 p = 0
Number "significantly" high 1
Number "significantly" low 2
Number "significantly" different 3
35
284427
925
164
687150
319
172
201633
165314
141
102
98
299102141
101134152
62
68
30
Site median
50
55
Median age
60
69
113
40
77
55
72
51
Services
49
Demographics: summary
All differences p<.0005
Quite large in magnitude
Number “significantly” different from
overall proportion/median ranged from 3
to 16 of the 33
Particularly big differences on ethnicity
Some of these demographic variables will
have relationships to outcome and failure
both within and between services
Individual level & site level effects
0.0
0.5
1.0
Sitewise correlations: age vs. CORE-OM non-risk improvement
45
326339
311
59
-0.5
74
25
45
61 67
98
84
127
139108
43
30
35 37
13
16
12
-1.0
8
Services
21
58
57 43
31
43
11
22
31
91
Overall correlation = 0.08
From n = 2510 n(miss) = 4100 %(miss) = 62
Maximum = 0.42
Minimum = -0.4
Number "significantly" high 1
Number "significantly" low 1
Number "significantly" different 2
Dotted CI indicates %(miss) > 50%
Starting points (1): on medication
135
141 94
49
75
51
Services
950
448
1564
1040
1805
1086
1823
40
173
151
299
681
509
125
101
1657
1744
1379
3008
812
1232
40
2
203
203
158
619
316309
121
413
141
77 68
121
227
20
1819
387
2021
5
1308
1496
329
968
1
7
1087
927
1389
915
4
60
%
532
1385
Overall proportion = 48.5%
From n = 5941 n(miss) = 669 %(miss) = 10
Higest proportion = 75%, lowest = 34.7%
Ratio, max:min = -1.05
Chi square = 141.29 d.f. = 32 p = 0
Number "significantly" high 6
Number "significantly" low 5
Number "significantly" different 11
Dotted red CI indicates %(miss) > 20%
1362
80
% on psychotropic medication
69
111
99
96
61
56
52
Starting points (2): CORE-OM score
532
2.0
1389
99 60
62
20
132
1040
448
2021
5
1232
387
1657
1
950
1744
1308
329
1564
1362
4
160
93
71
7
812
542
291
164
246
100
1379
3008
968
1496
1819
1805
1086
915
1087
2
343
853
1.5
Site median
203
927
1385
2.5
1823
Initial CORE-OM non-risk score
278
211
114
101
92
119134
60
132
100
638 66
187
48
56
33
Overall median = 2.14 on range from 0.04 to 4
From n = 5766 n(miss) = 844 %(miss) = 13
Highest median = 2.39 lowest = 1.46
Ratio, max:min = 1.64
Kruskal-Wallis chi square = 123.63 d.f. = 32 p = 0
Number "significantly" high 4
Number "significantly" low 4
Number "significantly" different 8
Dotted CI indicates %(miss) > 20%
107
Services
54
Starting points (3): % > CSC cut point
853
70
160
1389
93
164
99
132
134100114101
2021
448
5
1657
950
1232
343
187
132
100
66
119
92
54
60
71
56
60
48
62
33
Overall proportion = 79.5%
From n = 5766 n(miss) = 844 %(miss) = 13
Higest proportion = 89.4%, lowest = 49.5%
Ratio, max:min = 1.64
Chi square = 107.65 d.f. = 32 p = 0
Number "significantly" high 2
Number "significantly" low 3
Number "significantly" different 5
Dotted CI indicates %(miss) > 20%
20
50
60
291211
542
246
40
387
638
278
%
1819
1805
7
329
1744
1308
3008
1
1040
927
1823
1496
2
915
4
812
968
1379
1564
1087
1086
532
203
1385
90
80
1362
100
% > CSC cut point
107
Services
Starting points: summary
 All statistically significant p<.0005
 Large differences
 Number “significantly” different from overall
proportion/median ranged from 6 to 10 of the 33
 Again, starting conditions can have relationships with
outcome and failures at both individual and service
level
Logistics (1): wait time to assessment
418 58 61 17
128
177399
113
72 118
63 39
3
Services
329
4
1744
387
1362
1086
915
1805
3008
968
1823
7
927
1040
1308
5
2
1389
1379
1087
1232
2021
1
1819
448
532
50
94 378
101
26
216
168
0
77 76
149113
47 612
1564
1657
1496
Overall median = 31 on range from 0 to 583
From n = 4640 n(miss) = 1970 %(miss) = 30
Highest median = 137 lowest = 0
Ratio, max:min = Inf.
Kruskal-Wallis chi square = NA d.f. = 32 p = NA
Number "significantly" high 8
Number "significantly" low 11
Number "significantly" different 19
Dotted CI indicates %(miss) > 20%
950
Site median
100
203
150
1385
Wait time to assessment
58
317
90
64
198
101
89
Logistics (2): % offered more sessions
300
232
102
113
101
69
142 98
49
77
75
%
62
70
141
Overall proportion = 88.4%
From n = 6053 n(miss) = 557 %(miss) = 8
Higest proportion = 100%, lowest = 68.8%
Ratio, max:min = Inf.
Chi square = 248.69 d.f. = 32 p = 0
Number "significantly" high 6
Number "significantly" low 7
Number "significantly" different 13
Dotted CI indicates %(miss) > 20%
60
69 55
153135
128
1564
5
1362
1657
387
1308
1805
1744
2021
1379
203
1496
968
950
1232
927
329
812
4
7
1086
2
532
1
1819
448
425
521
124
102
3008
1040
1823
1389
80
121 60
690639
315
323
173165
203
1087
1385
90
915
100
% offered more sessions
51
50
40
Services
Logistics (3): #(sessions planned)
10
329
Number of sessions planned
1564
1385
8
203
1086
1362
270
145
79 94 238163 43 52 81 51 147391 64
71
448
121
34
532
65 14
151
24
Services
3008
1823
554 74 125 54 80 86 91
269
4
1819
1805
1744
1657
1496
1389
1379
1308
1232
1040
968
950
927
915
812
387
7
5
4
2
1
6
2021
50
2
Site median
1087
32 39
60
18
Logistics: summary
 All p<.0005
 All large differences, particularly for waiting time from
referral to assessment (13 days cf. 137 days)
 Number “significantly” different from overall
proportion/median ranged from 6 to 19 of the 33
 There are big differences on number of sessions offered
(medians from 3 to 10)
 … but many services offering fixed number, mode is six
sessions
 Looks very likely that there will be some differences
between services in the ways they operate that will
hugely affect outcome and failures
Outcomes (1): unplanned endings
533
102 67 76
85 56
49 44
99 93
37
26
82
0
133 39
Services
448
1385
2021
532
1496
950
3008
2
1805
1040
585
125 77 90 69
166169
260
122
1819
387
1232
1086
1362
1
1744
5
203
1087
1308
927
240
20
329
1657
7
1389
812
4
915
968
1379
40
1564
%
1823
Overall proportion = 38.5%
From n = 4396 n(miss) = 2214 %(miss) = 33
Higest proportion = 65.6%, lowest = 17.3%
Ratio, max:min = 3.33
Chi square = 162.78 d.f. = 32 p = 0
Number "significantly" high 6
Number "significantly" low 4
Number "significantly" different 10
Dotted CI indicates %(miss) > 20%
60
80
100
% offered more sessions
279
32
70
178 90
278
25
20
Outcomes (2): CORE-OM change
Outcomes (2): CORE-OM change
340
74
72
25
43
61
43 91
98
13
31
129
37
58 84
45
108
59
31 35
43
-2
16
11
-3
Overall median = -1.09 on range from -3.68 to 2.18
From n = 2519 n(miss) = 4091 %(miss) = 62
Highest median = -0.5 lowest = -1.39
Ratio, max:min = 0.36
Kruskal-Wallis chi square = 112.44 d.f. = 32 p = 0
Number "significantly" high 5
Number "significantly" low 3
Number "significantly" different 8
Dotted CI indicates %(miss) > 50%
8
Services
1496
1564
1389
1086
915
1040
2021
57
21
22
12
1819
3008
329
1823
950
1657
1744
387
Site median
968
812
7
203
448
927
1805
1
532
4
1308
1232
-1
5
2
326
139
312
1379
1385
1087
0
1362
Change on CORE-OM non-risk score
45
30
Outcomes (3): % RC
%
98 84
448
532
5
59
12
72
139
31
58
91
61 57
108
129
1308
927
4
1805
1232
1744
2021
203
2
74
312
327
60
7
968
950
1
1819
1362
1657
387
1086
812
1379
3008
329
1040
1389
915
340
1496
80
1564
1385
1087
100
1823
% w ith reliable improvement
25
43
11
21
43
37
45 43
31
22
Overall proportion = 78.4%
From n = 2520 n(miss) = 4090 %(miss) = 62
Higest proportion = 100%, lowest = 53.3%
Ratio, max:min = 0.36
Chi square = 76.97 d.f. = 32 p = 0
Number "significantly" high 3
Number "significantly" low 4
Number "significantly" different 7
Dotted CI indicates %(miss) > 50%
16
35
40
30
13
8
45
Services
Outcomes (4): % CSC
40
91 108
129
72 61
45 43
43 45
35
37
22
12
11
30
21
31
5
927
1308
1805
1744
7
2
1232
1819
532
1
968
950
1823
387
3008
1379
1657
203
4
812
915
329
1496
1385
1040
1362
1389
1564
2021
80
60
1086
58
327
16 13
Services
57
59
340
139
312
98 84
20
%
448
100
1087
% w ith clinically significant improvement
74
8
43
31
25
Overall proportion = 58.8%
From n = 2520 n(miss) = 4090 %(miss) = 62
Higest proportion = 87.5%, lowest = 32.3%
Ratio, max:min = 0.36
Chi square = 63.72 d.f. = 32 p = 0.0007
Number "significantly" high 2
Number "significantly" low 2
Number "significantly" different 4
Dotted CI indicates %(miss) > 50%
Outcomes: summary
All statistically significant p<.0005
Large differences
Number “significantly” different from
overall proportion/median ranged from 4
to 9
Despite large differences on RC and
CSC, number of services differing
“significantly” from the overall is not so
high (4 and 6 respectively)
Can automation of data processing
help bridge the gap?
 Neither researchers nor practitioners know much about
the generalisability of “strong causal inference” to
routine practice
 Need practice to come out of the confidentiality closet
without harming true confidentiality
 Very, very few services currently collect routine
outcome data
 Few services link with other services to compare
practices and data
 Few services have strong links to researchers to help
understand data
 Need to bridge these gaps: if we make data easier to
handle it might help!
Automation (1): batch route
Facilitates some distancing from the data
Data analyses done by researchers and
experts in analysis and data handling
Reports (30+ pages) well received
Can explore site specific issues
Automation (2): CORE-PC
“The clinical and reliable change graph is invaluable. As a service manager it
gives me instant access to where we can look to improve our service provision”
Automation (2): CORE-PC
“I never realised that
writing a report could be so
simple, all I need to do is
copy the tables I need from
CORE-PC, paste them in
Word, and write my
interpretations.”
Automation (2): PC
Allows services to get much “nearer” to
their data
Should prevent some data entry errors
Should increase data completeness
May mean that service clinicians and
managers feel uncertain about how to
analyse and interpret their data…
… will need training and support
http://www.psyctc.org/stats/Weimar
Not until Monday 30.vi.03!