Comparing the Reading
Demand of the English
CSAT and English
Textbooks in the
Republic of Korea
Laura Bush, Research Specialist; Heather Koons, Ph.D., Director of Research
Services; and Eleanor E. Sanford-Moore, Ph.D., Senior Vice President, Research
and Development; Special acknowledgement to Sally Ahn, International Marketing
Manager
ABSTRACT
The study examined the text complexity of passages from the
Republic of Korea’s English College Scholastic Ability Test
(CSAT) administered in 2015. The results were then compared
to the findings of a previous study of primary and secondary
English textbook complexity. The primary research question
investigated was whether text complexity varied from secondary
school, especially Grade 12, to the English College Scholastic
Ability Test.
BACKGROUND
Historically, Koreans have placed great importance on
education, an effect of the adoption of Confucian formal
teachings over fifteen centuries ago. Confucian beliefs hold
that through education, one may grow toward perfection and
that only those who reach the highest levels of education
should govern and lead society (Kwon, Lee, & Shin, 2015). For
hundreds of years, if a job candidate wanted to secure a good
position within the Government, and the associated wealth that
came with the position, he had to study long and hard to pass
state examinations (Choi, 2014). This desire for recognition
through erudition is still present in modern Korea and drives
the extensive spending on private education.
The trajectory of most primary and secondary education in
Korea culminates with the College Scholastic Ability Test
(CSAT) taken by high school seniors.1 In 2015, approximately
630,000 pupils took the high-stakes CSAT with hopes of
securing entrance to the university of their choice. The CSAT
tests pupils in the three mandatory subjects of Korean
language, English language, and mathematics, as well as
several self-selected subjects in social studies, the sciences,
and other foreign languages.
In 2015, 68.8 per cent of primary and secondary school pupils
were engaged in some form of private education, and 40 per
cent of these pupils participated in English-subject activities
(Statistics Korea, 2015). These activities include attendance at
after-hours intense study schools known as hagwons, group
and individual tutoring, Internet courses, and textbook studies
with tutor visits. Across all subjects, the greatest average
monthly expenditure per pupil in 2015 was for English
(Statistics Korea, 2015). Because wealthier families are more
able to spend money on this extra education for their children,
educational inequality driven by income levels has been a
concern.
The Educational Broadcasting System (EBS) is the only
educational television and radio company operating in the
Republic of Korea. While EBS operates mainly through private
revenue, it also receives annual grants from the Ministry of
Education to further its operations. In an effort to curb the
excessive spending on private education and create a more
equitable education system for all pupils, the Ministry of
Education has required a close link between the economical
educational materials produced by EBS (e.g., television
lectures, online activities, books) and the CSAT (MOE, 2004).
The content of the English CSAT is based on English I and
English II instructional materials approved for use by the
Ministry of Education in Grades 10 and 11 (KICE, 2016). It
follows that the content of the CSAT prep books would also be
based on the instructional materials for English I and English II.
In 2009, the government announced that 70 per cent of CSAT
items would be based on EBS materials (KICE, 2009). Item
writers alter EBS practice items for inclusion in the CSAT.
According to Kwon, Lee, and Shin (2015), this practice may
create outcomes other than reducing private expenditures on
education such as a reliance on EBS materials, problems
controlling CSAT item difficulty, and the inadvertent
encouragement of poor test preparation methods . In 2015,
EBS published four English CSAT test preparation books.
Examining the text complexity of the materials used for test
preparation is a means of understanding the English demands
presented to pupils at these critical stages in their education. In
addition, it provides a way of evaluating the connection
between the instructional materials and the assessment.
A recent MetaMetrics’s paper, Primary and Secondary English
Textbook Complexity in South Korea, examined the overall text
complexity (in Lexile measures) of Ministry of Education
approved, English subject materials for state-maintained
primary and secondary schools (Grades 5 through 12). The
findings suggested that, beginning in Grade 5, the English text
complexity that Korean pupils encounter increases
monotonically (median Grade 5, 140L; median Grade 12,
1180L). However, the range of complexity found at each grade
varies substantially (Sanford-Moore, Koons & Bush, 2016). The
current study compares the complexity measures of the
previously studied Grade 12 materials with those from the most
recent English CSAT. The expectation is that Grade 12
materials and the English CSAT passages have similar levels of
complexity as measured by The Lexile Framework for Reading.
Typically high school pupils take the CSAT during their senior year;
however, pupils may take the CSAT again by waiting for the next
administration in an effort to increase their scores.
1
MATERIALS
ANALYSIS
The English CSAT is composed of both listening and reading
items. The units of analysis in this study were passages from
the 2015 English CSAT reading section with enough English
prose to measure for text complexity (N = 25). For purposes of
comparison, tables and figures refer to the previous study of
primary and secondary English textbooks which focused on
reading passages within each textbook and grade (SanfordMoore, Koons & Bush, 2016). Grade 12 materials included 755
practice English reading CSAT passages.
For each source, descriptive statistics for the passage Lexile
measures were calculated. The percentiles of the text
complexity measure distributions provided the basis for
constructing the box-and-whisker plots featuring the 5th,
25th, median (50th), 75th, and 95th percentile values of the
distribution. The box-and-whisker plots are displayed in
Figure 1. Finally, the means and standard deviations were
calculated to determine the overall text demand.
PROCEDURE
Only the English test items that could be categorised as a passage (continuous English prose) were analysed. For cloze
items, an answer choice was selected (A, B, C or D) in a rotating basis throughout the test in order to create complete sentences. Each passage was converted to electronic text and
prepared for analysis. All non-prose text was removed. The
electronic texts were then submitted to the Lexile Analyzer and
each assigned a Lexile text measure.
MEASURES
The Lexile® Framework for Reading is a scientific way to match
pupils with text using the same developmental scale. Lexile text
measures (Stenner, H. Burdick, Sanford & D.S. Burdick, 2007)
are measures of text complexity that are based on semantic
and syntactic factors. Independent psychometric studies of the
Lexile scale indicate that it is a valid and reliable measure of
pupil ability and text complexity (Mesmer, 2008; White & Clement, 2001).
A Lexile measure is the numeric representation of an individual’s reading ability or a text’s complexity (or, difficulty) followed
by an “L” (for Lexile). The Lexile scale is a developmental scale
for measuring pupil ability and text complexity, ranging from
below 200L for beginning pupils and beginning reading materials to above 1600L for advanced pupils and materials. Extensive information about the development of The Lexile Framework for Reading can be found in the “Research and Publications” section of the Lexile website (www.Lexile.com/researchand-publications).
RESULTS
Table 1 contains the 25th percentile, median, and 75th
percentile Lexile measures for the 2015 English CSAT as well
as the Grade 12 data from the previous study. Figure 1
displays box-and-whisker plots for the 2015 English CSAT as
well as the textbook passages for Grade 12. The lower box
represents the 25th to 50th percentile while the upper box
represents the 50th to 75th percentile of text measures for
each group. The ends of the whiskers represent the 5th and
95th percentiles for each source. The boxes represent 50 per
cent of the passage measures. The data indicates that the
median Lexile measure for the English CSAT is slightly lower
(70L) than the median Lexile measure for the Grade 12 CSAT
prep materials.
An analysis of variance (ANOVA) revealed no statistically
significant difference between the text complexity means of
the Grade 12 and the English CSAT materials at the p < .05
level; F(1, 778) = 0.52, p = 0.47.
DISCUSSION
The findings indicate that Grade 12 pupils encountered
reading material in the 2015 English CSAT of comparable
reading challenge to that of the CSAT prep books. These
results support the Ministry of Education’s goal of creating an
equitable education system for all pupils in the Republic of
Korea through the development of readily available
educational materials. While the findings indicate that the
CSAT contains items of similar text complexity to the EBS
preparation materials, questions about unintended negative
effects remain. However, given the extreme financial burden
of private education in Korea, the availability of affordable test
preparation materials may outweigh these possible negative
effects.
Table 1. Median and interquartile range of passage text complexity measures.
Source
25th Percentile
Median
75th Percentile
Grade 12 CSAT Prep Books
1030L
1180L
1310L
2015 English CSAT
1000L
1110L
1240L
Figure 1. Text complexity distributions
2000
1900
1800
1700
1600
1500
Lexile Framework
1400
1300
1200
1100
1000
900
800
700
600
500
400
300
200
100
0
Grade 12
CSAT 2014
References
Choi, A. (2014). What the best education systems are doing right. Ideas.TED.Com. Retrieved from: http://ideas.ted.com/what-the-best-education-systems-are-doing-right/
Korea Institute for Curriculum and Evaluation (KICE). (2016). “College Scholastic Ability Test.” Retrieved April 26, 2016 from http://www.kice.re.kr/sub/info.do?m=0205&s=english
Korea Institute for Curriculum and Evaluation (KICE). (2009). Announcement of CSAT Basic Plan for the School Year. http://www.suneung.re.kr/boardCnts/view.do?
boardID=1500230&boardSeq=5005133&lev=0&m=0302&searchType=null&statusYN=W&page=1&s=suneung
Kwon, S. K., Lee, M., & Shin, D. (2015). Educational assessment in the Republic of Korea: Lights and shadows of high-stake exam-based education system. Assessment in Education:
Principles, Policy & Practice. Retrieved from http://dx.doi.org/10.1080/0969594X.2015.1074540
Mesmer, H. A. (2008). Tools for matching readers to text: Research based practices. New York, NY: The Guilford Press.
Ministry of Education (MOE). (2004). Reduction of Private Education Expenditure. Retrieved from http://english.moe.go.kr/web/1707/site/contents/en/en_0275.jsp
National Curriculum Information Center (NCIC) (no date). “National Curriculum of Korea.” Retrieved April 26, 2016 from http://ncic.re.kr/
english.kri.org.inventoryList.do;jsessionid=A59A8A0BB6E5CA3353F424A880538671
Statistics Korea. (2016). Private Education Expenditures Survey in 2015, 2.25.2016. Retrieved from http://kostat.go.kr/portal/eng/pressReleases/1/index.board?
bmode=read&bSeq=&aSeq=352520&pageNo=1&rowNum=10&navCount=10&currPg=&sTarget=title&sTxt=education
Sanford-Moore, E., Koons, H. & Bush, L. (2016). Primary and Secondary English Textbook Complexity in the Republic of Korea. Retrieved from https://metametrics.s3.amazonaws.com/
public/dynamic/international/pdfs/MM_Textbook_complexity_in_Korea_Digital.pdf
Stenner, A. J., Burdick, H., Sanford, E. E. & Burdick, D. S. (2007). The Lexile Framework for Reading Technical Report. Durham, NC: MetaMetrics, Inc.
White, S. & Clement, J. (2001). Assessing the Lexile Framework: Results of a panel meeting. NCES Working Paper Series, Working Paper No. 2001-08. Washington, D.C.: U.S. Department
of Education, Office of Educational Research and Improvement.
For more information contact [email protected]