Further Reactions to “First Reactions to the Programme for the International
Assessment of Adult Competencies (PIAAC) Results”
Brigid Hayes
January 6, 2014
On October 8, 2013, the OECD released the results of the International Assessment of Adult
Competencies (PIAAC). I provided a commentary on those results, First Reactions to the
Programme for the International Assessment of Adult Competencies (PIAAC) Results 1,
sponsored by the Canadian Literacy and Learning Network.
Following the posting of that paper, I received various comments, including several
provided at the Centre for Literacy’s Fall Institute, Interpreting PIAAC Results:
Understanding Competencies of the Future, in late October. After the Fall Institute, I
received clarification from Employment and Social Development Canada (ESDC) about
points I had made. In addition, having time for further reflection, I wanted to share
additional thoughts about PIAAC. As always, these views are my own and this particular
paper is not sponsored by any organization.
Follow-up from Employment and Social Development Canada 2
In my initial reaction, I pointed out differences between the OECD report, OECD Skills
Outlook 2013 First Results from the Survey of Adult Skills, and the Canada report, Skills in
Canada: First Results from the Programme for the International Assessment of Adult
Competencies (PIAAC). In doing so, I suggested that the Canada report was lacking and
implied that this was a reflection of a policy stance.
Differences between the OECD and Canada report
ESDC graciously provided me with information about the process leading up to the
publication of the Canada report that provides a context for the differences between the
two reports. I had forgotten that for the 2005 International Adult Literacy and Skills Survey
(IALSS), the Canada report was released months after the international report, while this
time, the OECD and PIAAC reports were released on the same day. The OECD had 18
months to examine the data and prepare its report. ESDC had the data for 8 weeks and
“October 9: Brigid Hayes: First Reactions to the Programme for the International Assessment of Adult
Competencies (PIAAC) Results.” http://www.literacy.ca/research-and-resources/ and « Premières réactions
aux résultats du Programme pour l’évaluation internationale des compétences des adultes (PEICA) »,
http://resdac.net/forumpeica/
2 Thank you to Patrick Bussière, Director, and Bruno Rainville, Manager, Skills Development Research
Division, ESDC, for their assistance in setting me straight.
1
-1-
chose to present a broad overview. With various provincial and territorial partners and the
need for a bilingual text, the timelines were very tight. While it would have been helpful to
have had this information about the timelines and choices in presentation in the Canada
report, now that I am fully aware, I wanted to share this information and correct any
misperceptions my original piece may have left.
Additional Reports
The title of the Canada report was “First Results.” I wondered in my original paper what
else might happen in the future. ESDC indicated that the ‘first results’ dealt only with the
three main domains – literacy, numeracy and problem solving in technology-rich
environments (PS-TRE). Further analysis will be provided in a series of PIAAC thematic
reports produced by the federal, provincial, and territorial government partners, over the
next two years. Topics include:
1.
2.
3.
4.
5.
6.
Skills and labour market outcomes (including skills mismatch)
Skills of the immigrant populations
Skills and education
Skills of official-language minority populations
Skills of the aboriginal populations
Skills and health and social outcomes
“PIAAC in focus” are shorter reports with a particular focus and provide quick facts on
topics of interest. The current plan is to prepare three PIAAC in focus:
1. Skills and Employment and Earnings
2. Skills in the North
3. Skills, Education and Aboriginal populations
In addition, the OECD will release its own reports over the next two years:
1.
2.
3.
4.
5.
6.
Skills and labour market outcomes
The use of skills in the workplace
Skills mismatch
A closer look at the population with low levels of proficiency
Digital literacy, problem solving in Technology Rich Environments and using ICTs
Trends, ageing and the determinants of skills
-2-
Data Presentation
As I mentioned in the original piece, the data itself is available via the OECD website, PIAAC
International Data Explorer, the International Database (IDB) Analyzer or the Canadian
public use data-file. This will reduce the dependency on government organizations for
additional analysis.
Because the data is available through these sources and to avoid redundancy, ESDC chose
to present only one type of analysis, e.g. average scores rather than both average scores
and proficiency by levels. I would still suggest that not having proficiency scores by levels
for many of the demographic elements, as had been the case in the previous reports, makes
it hard to get a good picture. While I understand the need to make choices, relying on
individuals to do their own analysis is not ideal and I would hope that ESDC could produce
some fact sheets with both average scores and information by proficiency levels.
Combining Prose and Document Literacy
A question that arose after the positing of my original piece was about the combining of
prose and document literacy into a single measure called “literacy.” Many who are engaged
in workplace literacy and essential skills efforts use the concept of document literacy to
explain the differences between school-based literacy and workplace literacy. Apparently,
the PIAAC Board, made up of participating countries, took this decision. Despite reviewing
references provided by ESDC, I have not been able to find an explanation for this decision.
Reading Components – Below Level 1
PIAAC administered a reading components assessment to those whose skills fell below
level 1. While this was a welcomed addition in terms of information, I expressed concerns
about the test itself which, at first blush, appeared to be the same as the International Study
of Reading Skills (ISRS). That survey, administered in 2005, tested word recognition,
vocabulary, basic text processing, and spelling. Concerns were raised at that time about the
tests themselves many of which were designed for children 3. I did some more research on
the PIAAC reading components assessment and found that it is not the same as the ISRS
except for the area of vocabulary 4. The OECD is doing further work on this topic and I hope
that Canadian researchers will take a closer look at the below level 1 group (there will be
no official Canadian thematic report on the topic).
3
4
literacies, Special Bulletin, October 2008. “Reading the Reports”.
OECD. The Survey of Adult Skills. Reader’s Companion. Page 78.
-3-
Problem Solving in Technology-Rich Environments (PS-TRE)
A new element was added to PIAAC – Problem Solving in Technology-Rich Environments
(PS-TRE). This was an effort to understand “using digital technology, communication tools
and networks to acquire and evaluate information, communicate with others and perform
practical tasks.”
Initially I commented that,
…all of the sample questions require knowledge of email and spreadsheets, for
example, sorting emails into folders. For adult Canadians not in school, these skills
would have likely only have been developed in a work situation. This means certain
individuals may have scored lower since they had not had the chance to develop
these particular skills at work, even if they had the computer skills.
The lively discussion at the Centre for Literacy’s Fall Institute confirmed some of my
concerns about PS-TRE. People wondered about the use of keyboarding technology in an
era of swiping and gestures. The actual laptops and programs used for the survey were
created at least five years ago, a lifetime in the fast-paced technology evolution. The
exclusion of those who could not use a computer was a singular criticism raised and begged
the question of assessing problem solving skills that do not use technology. PS-TRE is being
discussed in various webinars and will be a key topic in the Centre for Literacy’s Summer
Institute in June 2014.
Reactions
The Centre for Literacy compiled information on media reactions to PIAAC. Its assessment
was “Selected reaction from media and commentators to the release of PIAAC results
ranged from muted and mixed in Canada, to minimal in the US where the federal
government shutdown prevented release of the national report, to alarmed in England
where young people showed no greater skills than people over 55.” 5
The Council of Ministers of Education (CMEC) issued a press release as well as hosting a
website on PIAAC. Several provincial and territorial governments also issued comments
about the PIAAC results. Most appear to be taking a measured response by promising to
study the report before taking action.
Centre for Literacy. “Early analyses of response to PIAAC.” http://www.centreforliteracy.qc.ca/news/earlyanalyses-responses-piaac#intl.
5
-4-
The only federal news announcement about PIAAC was Statistics Canada, The Daily. 6 ESDC
did not issue a statement nor can you find any reference to the PIAAC report in an online
search of the ESDC website.
The silence at the federal level is disturbing. This may be consistent with the recent
proposed changes to the Labour Market Agreements which would see the end of literacy
and essential skills as a priority area. The federal government spent millions on PIAAC and
encouraged provincial and territorial counterparts to do the same. Ignoring the policy
implications of PIAAC should not be acceptable.
Positive Aspects of PIAAC
•
•
•
•
•
•
•
Provides a benchmark against which Canada can assess its performance
Maintains a focus on literacy, PIAAC will also provide greater information about
those at level 1 and below
Links literacy to issues of social well-being, inequality, and success planning
Rich source of information (still to be mined) on the role of literacy skills at work
and literacy practice
Will also enable an exploration of qualifications vs. skills
Links to PISA (Programme for International Student Assessment) involving all 15
year olds (how to unhook parental education from child’s prospects)
Public access to data for further research
Some Questions and Challenges
•
•
•
The federal government has taken a low profile on PIAAC – not clear why or what
the impact will be
While OECD has rejected level 3 as a benchmark, many jurisdictions tied their
policies to achieving level 3 – it remains to be seen what they will do next
The OECD has a strong skills perspective with literacy being but one of a series of
skills – will need to continue to look at literacy as a social as well as economic good
6 Statistics Canada. The Daily, October 8, 2013. http://www.statcan.gc.ca/dailyquotidien/131008/dq131008b-eng.htm
-5-
•
•
The merging of prose and document literacy may deter employers from
understanding “literacy” as a workplace issue
The implications of the PIAAC domains for the Essential Skills Framework,
especially PS-TRE and the essential skills of critical thinking and computer use
Take-away’s
•
•
•
•
•
•
PIAAC provides important information
It is an assessment of populations, not a curriculum
It measures literacy in ways that are conducive to measurement
IALSS led to a call to ‘move people to level 3’ – PIAAC can be used in a more nuanced
fashion
Analysis from independent non-government groups will help balance the OECD and
government’s perspectives
Need to find a variety of ways to look at literacy – not just quantitative indicators –
what are those indicators, how do we measure them, how is the story told.
The comments in this paper are mine alone. I welcome your comments and feedback.
Brigid Hayes, Brigid Hayes Consulting, [email protected]
-6-