Uneven language
proficiency: How
spiky can a spiky
profile be?
Karen Smith
Lecturer EAP/Study Skills
University of Central Lancashire,
Preston
Overview
• Research questions:
• What is the shape of uneven proficiency in EAP candidates across the 4 skills?
• Are there correlations between the skills?
• What are ‘spiky profiles’ and how ‘spiky’ can they get?
‘spikiness’
Should we expect big score
differences across the skills?
How much
variation is
possible or
natural?
Should we
expect some
skills to be
always stronger
than others?
Does variation
matter?
Are the 4 skills connected?
• In L1 listening and speaking develop together
• Then ‘literate’ skills grow from speaking & listening
• Writing skills grow from speaking & listening (Berninger & Abbot,2010)
• The same brain mechanism is used for syntactic encoding in speaking
and writing (Cleland & Pickering, 2006)
• Speaking does not always precede writing in L2
• Individual differences have an impact: modality strength, preference,
cognitive style, affective factors (Weissberg, 2006; Kormos & Trebits
2012; Baba,Takemoto & Yokochi 2013)
UCLan’s TELL
• Listening (30 mins) 3 different audio
recordings, diff types of interaction:
informal, semi-formal and formal
• 20 marks
• Reading (55 mins) range of texts, diff.
skills tested
• 30 marks
• Writing (45 mins) essay writing task of
180-200 words (B2) 250-280 (C1)
• 20 marks (10 bands)
• Listening & reading marked optically
• Min 20% of all scripts 2nd marked
• Speaking
• 11 mins (B2) 13 mins (C1)
• 3 parts: intro, interactive
discussion, responding to
questions
• Grammar, vocab, pronunciation,
discourse management,
interactive ability
• 30 marks (5 bands)
Exam Data from the TELL
• B2 & C1 levels under investigation
• December 2016 sittings in 4 centres: Preston, Sunderland, Hull &
Coventry
• 542 B2 candidates’ l/r/w/s scores compared
• 56 C1 candidates’ l/r/w/s scores compared
• Mixed nationalities, all adults, 21-25 years old
One-upmanship among the skills?
B2 data n=542
C1 data n=56
Speaking better than
Writing
Reading
Listening
96.9%
(0.7% same)
100%
79.5%
(2% same)
84%
(7.1% same)
70.8%
(2.4% same)
71.4%
(1.8% same)
Listening better than
84.5%
(7.2% same)
96.4%
(1.8% same)
63.5%
(3.3% same)
78.6%
Reading better than
76.9%
(2.4% same)
94.6%
(1.8% same)
Writing better than
20.7%
(2.4% same)
3.6%
(1.8% same)
Speaking
26.8%
(2.4% same)
26.8%
(1.8% same)
33.2%
(3.3% same)
21.4%
18.5%
(2% same)
8.9%
(7.1% same)
8.3%
(7.2% same)
1.8%
(1.8% same)
2.4%
(0.7% same)
0%
Spearman’s rho correlation of each skill with overall score
Skills
Type
B2 (n=542)
rs
Reading & Overall
Receptive Literate
.890 (79%) L
.849 (72%) L
Listening & Overall
Receptive Innate
.860 (74%)L
.642 (41%) L
Speaking & Overall
Productive Innate
.797 (64%) L
.715 (51%) L
Writing & Overall
Productive Literate
. 785 (62%) L
.841 (71%) L
C1 (n=56)
December 2016 Skills’ correlations at B2 & C1
Skills
Type
B2 (n=542)
R
Coefficient of determination
in brackets
B2 (n=542)
rs
C1 (n=56)
rs
Listening & Reading
Receptive
.698 (49%)L
.706 (50%)L
.316 (10%) M
Reading & Writing
Literate
. 654 (43%)
.662 (44%)
. 794 (63%) L
Speaking & Listening
Innate
. 574 (33%)
.594 (35%)
. 557 (31%) L
Speaking & Reading
. 556 (31%)
.571 (33%)
. 459 (21%) M
Listening & Writing
. 524 (27%)
.546 (30%)
. 282 (8%) S
.518 (27%)
.523 (27%)
. 374 (14%) M
Speaking & Writing
Productive
Quartiles of the overall score at B2
Overall
score
% of
sample
mean
score
Speaking
mean
score
Listening
mean
score
Reading
mean
score
Writing
S-W% L-W%
gap
gap
R-W%
gap
S-R%
gap
S-L%
gap
L-R%
gap
0-25
0
26-50
20.3
59.32
45.41
37.55
35.77
23.55 9.64
1.78
21.77
13.91
7.86
51-75
55.5
75.12
68.55
61.24
47.76
27.36 20.79
13.48
13.88
6.57
7.31
76-100
24.2
92.92
85.76
84.44
65.99
26.93 19.77
18.45
8.48
7.16
1.32
A closer look at Writing vs. Speaking at B2
Speaking of
Mean % gap
scores
sample with writing
n=542 score
SD
Writing
scores
% of
sample
n=542
Mean % gap
with speaking
score
SD
0-25
0.2
35.00
0
0-25
3.4
44.11
14.50
26-50
3.3
10.22
6.99
26-50
61
29.20
13.66
51-75
45.2
21.16
10.62
51-75
31
22.91
12.17
76-100
51.3
32.97
14.29
76-100
4.6
9.80
6.48
CEG08035196
CEG08032062
CEG08036211
CEG08035452
CEG08034409
CEG08032941
CEG08032834
CEG08035363
CEG08034774
CEG08034560
CEG08036069
CEG08034984
CEG08032066
CEG08033739
CEG08029912
CEG08035050
CEG08035542
CEG08023854
CEG08032405
CEG08034289
CEG08035159
CEG08036084
CEG08034522
CEG08035118
CEG08036015
CEG08034894
CEG08031988
CEG08034294
CEG08034772
CEG08032998
CEG08034873
CEG08035317
CEG08036036
CEG08033735
CEG08034976
CEG08035560
CEG08033736
CEG08034929
CEG08035129
CEG08036050
CEG08035345
CEG08029256
CEG08033815
CEG08034712
CEG08035655
CEG08032632
CEG08034769
CEG08035327
CEG08036182
CEG08034503
CEG08035607
CEG08033587
CEG08031596
CEG08034182
CEG08035701
CEG08034525
CEG08034529
CEG08034254
CEG08032850
CEG08033826
CEG08035669
CEG08032227
CEG08034950
CEG08036126
CEG08034271
CEG08035657
CEG08033642
CEG08031737
CEG08022749
CEG08036127
CEG08036153
CEG08035088
CEG08036296
CEG08030011
CEG08033156
CEG08035825
CEG08035852
CEG08035269
Percentage weighted score
Writing vs. speaking scores B2 December 2016
120
100
80
60
40
20
0
Series1
Candidates n=542
Series2
A closer look at Reading vs. Listening
Reading
scores
of
Mean % gap SD
sample with listening
n=542 score
Listening of
Mean % gap
scores
sample with reading
n=542 score
SD
0-25
3%
26.13
11.92
0-25
1%
9.50
9.00
26-50
27%
14.85
11.08
26-50
20%
10.97
8.69
51-75
41%
12.46
9.25
51-75
45%
12.89
9.47
76-100
29%
8.46
6.18
76-100
34%
12.58
10.52
The picture so far…
• What are the parameters of uneven proficiency across the 4 skills?
•
•
•
•
•
Speaking appears to be the strongest, even in the weakest candidates
Writing is the weakest link for all (no surprise there!)
Writing is very rarely better than speaking (2.4% of candidates)
Reading also lets down the weakest
Top quartile writing scores show smallest gap with speaking
• Are there correlations between the skills?
• receptive skills (R&L) are the most correlated at B2, not C1
• literate skills (R&W) also strongly correlated, more at C1
• Despite both being productive, Speaking & Writing are least correlated
Further study in spikiness…
• Compare with other proficiency test data
• Closer inspection of the speaking and writing scores:
what is going on within the scores?
• Compare performance under exam conditions against
relaxed conditions (affective dimension)
• Explore modality preference & self-efficacy
(affective/personality-related)
References
Baba, K., Takemoto, Y. and M. Yokochi. 2013. ‘Relationship between second language speaking and writing skills and
modality preference of university EFL students’. Japanese Institutional Repositories Online. (Retrieved 14 March 2015
from http://jairo.nii.ac.jp/0263/00000548.)
Bereiter, C. & M. Scardamalia. 1987. The Psychology of Written Composition. Hillsdale, NJ: Lawrence Erlbaum
Associates.
Berninger, V. W., & Abbott, R. D. 2010. ‘Listening comprehension, oral expression, reading comprehension, and
written expression: Related yet unique language systems in grades 1, 3, 5, and 7’. Journal of Educational Psychology,
102/3: 635-651.
Cleland, A. and M. Pickering. 2006. ‘Do writing and speaking employ the same syntactic representations?’. Journal of
Memory and Language 54: 185 – 198.
Council of Europe. 2011. Common European Framework of Reference for Languages: Learning, Teaching, Assessment.
Language Policy Unit, Strasbourg. (Retrieved 14 March 2015 from
http://www.coe.int/t/dg4/linguistic/Source/Framework_EN.pdf.)
Dörnyei, Z. 2005. The Psychology of the Language Learner: Individual Differences in Second Language Acquisition.
Mahwah, NJ: Lawrence Erlbaum.
Kormos, J. and A. Trebits. 2012. ‘The role of task complexity, modality and aptitude in narrative task performance’.
Language Learning 62/2: 439 – 472.
Weissberg, R. 2008. ‘Critiquing the Vygotskian approach to L2 literacy’ in D. Belcher & A. Hirvela, A. (eds.). The Oralliterate Connection: Perspectives on L2 Speaking, Writing, and Other Media Interactions. Ann Arbor: University of
Michigan Press.
Williams, J. 2008. ‘The speaking-writing connection in second language and academic literacy development’ in D.
Belcher & A. Hirvela, A. (eds.). The Oral-literate Connection: Perspectives on L2 Speaking, Writing, and Other Media
Interactions. Ann Arbor: University of Michigan Press.