PART 1 - INVITATION TO QUOTE
Quotes are invited by the Australian Curriculum, Assessment and Reporting Authority
(ACARA) for a contract in relation to the NAPLAN Online trial – Cognitive interviews: Writing
August 2013 (as specified in this document).
Quotes are to be submitted on the basis that they conform to Part 2: Specifications and Part
4: Response forms.
Agency
Australian Curriculum, Assessment and Reporting Authority
Closing Date
5:00pm (Sydney time), Friday 19 April 2013
Contact Officers
For further information you can contact:
Peter Matheson on ph: (02) 8098 3116 or email [email protected]
John Skehan on ph: (02) 8098 3133 or email: [email protected]
Quote Validity Period
Quotes must remain valid for a minimum period of 90 days after the Closing Date.
Lodgement of Quote:
Quotes submitted in response to this Invitation will be accepted by ACARA via Email
You are required to email the full quote as a PDF to [email protected] with a
CC to [email protected] with the following text in the message box NAPLAN
Online Trial Study: Cognitive interviews Writing
PART 2 – SPECIFICATIONS
2.1
Introduction
The Australian Curriculum, Assessment and Reporting Authority (ACARA) is responsible for
the development of a rigorous, world-class Australian curriculum from Foundation to Year 12.
To complement the development of an Australian curriculum, ACARA is also responsible for
developing and administering a national assessment program aligned to the national
curriculum that measures students’ progress, and the provision of information, resources,
support and guidance to the teaching profession.
ACARA reports to, and is directed by, the Standing Council on School Education and Early
Childhood (previously known as the Ministerial Council for Education, Early Childhood
Development and Youth Affairs).
ACARA has overall responsibility for implementing the National Assessment Program (NAP).
2.2
Background
National Assessment Program – Literacy and Numeracy
The National Assessment Program — Literacy and Numeracy (NAPLAN) is the main
assessment program conducted by ACARA. Every year all Australian students in Years 3, 5,
7 and 9 are assessed in Reading, Writing, Language Conventions and Numeracy. The tests
are equated longitudinally each year in order to ensure comparability of results from year to
year, and to enable tracking of students’ results.
National Assessment and Surveys Online Program
The National Assessment and Surveys Online Program (NASOP), under the Australian
Government’s Online Diagnostic Tools election commitment, is intended to develop capacity
to deliver national assessments and surveys through an online environment.
This Program comprises three elements:

development, testing and delivery of a national online School Opinion Survey for
deployment in 2013 at jurisdictional discretion (Element 1),

development, testing and delivery online of the 2013 National Assessment Program –
Civics and Citizenship sample assessment (NAP-CC) (Element 2), and

planning and implementing a clearly defined assessment and reporting research agenda
that will allow reporting to SCSEEC on issues and options for delivering NAPLAN online
(Element 3).
NAPLAN Research Study
ACARA has developed a comprehensive research agenda that will provide an evidentiary
base for SCSEEC decisions concerning the transition of NAPLAN from a pen-and-paper test
to a computer-based assessment. The research agenda consists of three studies: a pilot
study, a trialling study and a linkage study.
ACARA completed the first stage, the pilot research study, in December 2012. The second
stage of the research study focuses on trialling the NAPLAN online test structure and data
collection, which will occur in August 2013. As part of this trial ACARA will be conducting a
study investigating options for future assessment of writing.
2.3
Purpose
The purpose of this contract is to engage a qualified and experienced Contractor to
investigate the cognitive and behavioural engagement of students with NAPLAN Writing
items and instruments delivered on computer using cognitive interview methodology
(cognitive interviews component).
The cognitive interview will cover two tasks each year level. The test design for Writing will be
confirmed by ACARA after project initiation.
This investigation will be conducted in conjunction with the NAPLAN Online Writing Trialling
Study utilising the same assessments and a sub-sample of the schools participating in the
Trialling Study.
The NAPLAN Online Writing Trialling Study will be invigilated by a third party, the NAPLAN
Online Writing Trialling Study Contractor, who will be administering the tests and who will
provide assistance with the test delivery platform.
2.4
Scope of project
The successful provider will be required to:

Conduct cognitive interviews as one-on-one sessions with a random sample of students
participating in the NAPLAN Online Writing Trialling Study primarily in metropolitan
schools but also in some regional schools. It is further expected that the students will be
recruited from at least three different schools. The anticipated sample size is 10 students
per test and per year level. Each student will complete two (2) writing tasks. Where
conditions allow and where valid observations could be obtained, two students could be
simultaneously interviewed and/or observed.

The cognitive interview will focus on capturing all the mental processes that the student
engages while completing test questions. Each student will be asked to provide some
initial response to viewing and navigating each prompt on-screen and then, on completion
of each writing task, invited to reflect on the experience of generating an extended
response on-screen.

Liaise with ACARA in the construction of the sample of schools and/or students that will
be included in this study. The schools will be sampled from those participating in the
NAPLAN Online Writing Trialling Study.

Participate in training on the online test delivery system which will be provided by
ACARA’s Test Delivery System Contractor.

Work with ACARA to finalise protocols for conducting the cognitive interviews, drawing on
existing protocols developed for ACARA in previous studies (see Appendix A). Methods
utilised in previous studies involved four aspects:
o
Concurrent observation– observing the student’s interaction with the items
o
Concurrent think aloud – using the “think aloud” method on pre-selected items
o
Concurrent interview – a structured question to the student immediately after they
have finished responding to an item
o
Retrospective interview– a series of structured questions about the student’s
experience with the items and the online test overall.
Students will be involved in two sessions. Session 1 will include the concurrent
observation, concurrent interview and “think aloud” aspects. Session 2 will consist of the
retrospective interview.

Observations/interviews are to be conducted by interviewers who have relevant
experience in cognitive interviewing as well as an understanding of the construct being
tested. The Contractor will provide ACARA with evidence of the interviewers’ experience
and qualifications.

It is expected that the Contractor will provide training specific to this project to ensure
interviews are conducted consistently by the interviewers. A combined video recording of
the screen and audio recording of each student is to be made for all sessions. Once all
interviews have been completed, the records should be collated into spreadsheets and
combined with the student response data from the online platform.

If electronically recording the interview the Contractor will be required to obtain consent
from parents/guardians of students that will be included in the study to interview the
students and use the information from the cognitive interviews for the purpose of the
study. ACARA will provide a pro forma consent agreement to be used by the Contractor.

The Contractor will liaise and coordinate with the:

2.5
-
Test Delivery System Contractor regarding the test delivery system
-
NAPLAN Online Writing Trialling Study Contractor regarding the times and dates
of cognitive interview implementation and communication to schools
-
Test administrators for the NAPLAN Online Writing Trialling Study on the day of
the tests regarding the specific students who will be participating in the cognitive
interviews.
Produce a report summarising the main findings of the cognitive study, particularly
regarding the engagement of students with the tailored test design. The Contractor is
required to obtain an accurate record of each interview either through electronic recording
or by transcribing the interview.
Timing
Note: The timeframe will be confirmed during contract negotiations, and the final
timeframe will be included in the Contract. Organisations should use the
following information as the basis for preparing their responses. Although
ACARA is prepared to discuss changes to the proposed timeframe, it reserves
the right not to alter the proposed timeframe.
It is expected that this project will commence in June 2013 and be completed by midNovember 2013.
Task name
Start
End
Training on Test Delivery System
June 2013
June 2013
Finalisation of school sample for cognitive
interviews
19-Jul-13
19-Jul-13
Finalisation of cognitive interview protocols and
scripts
26-Jul-13
26-Jul-13
Implementation of the cognitive interviews
12-Aug-13
6-Sep-13
Draft Report on cognitive interviews
27-Sept-13
7-Oct-13
Final Report on cognitive interviews
8-Oct-13
6-Nov-13
PART 3 – EVALUATION PROCESS & CRITERIA
3.1
Evaluation Process
The evaluation process will involve an assessment of quotes received against the criteria
listed below. ACARA reserves the right to short-list during the evaluation process using the
evaluation criteria contained below. The evaluation process may also involve discussions,
reference, financial and corporate checks.
3.2
Evaluation Criteria
Offers received will be evaluated against the following criteria (if applicable). The weightings
ascribed to each criterion will be applied consistently.

Ability to meet the Objectives/Scope outlined in the Specification

Reputation and Experience (track record, competence, capacity to deliver)

Ability to meet deadlines and conduct activities in a timely fashion

Value for money.
PART 4 – RESPONSE FORM
(The Applicant must complete all of the following Response Forms as part of its
Quote)
Response Form 4.1 - Applicant Authorisation and Certification
Information Required
Name of applicant:
 if applicant is a company –
specify the full company
name;
 if applicant is a trust – specify
the names of each trustee of
the trust;
 if applicant trades under a
business name, specify the
registered business name; or
 if applicant is an individual or
partnership, specify the full
name of each individual or
partner.
Applicant’s ACN (if applicable)
Applicant’s ABN
State or Territory in which Business
/ Corporation is registered (*)
Name of Holding Company /
Corporate Group (*)
Company Profile
Details
OR if no ABN – have you submitted a ‘Statement by
a supplier – Reason for not quoting an Australian
Business Number (ABN) to an enterprise’ form?
Yes
No
As part of the quote process, each applicant must
include a copy of their Company’s Profile.
Does your quote include your Company’s Profile:
Yes
No
Yes
No
Is the applicant registered for GST?
Postal Address
Street Address
Contact Name
Contact Phone
Contact E-mail
Contact Fax
Is it proposed to sub-contract any
Yes
No
part of the Goods and/or Services?
If “YES”, please specify full name
and address of each sub-contractor
and their relevant experience and
expertise in relation to the offered
Goods and/or Services.
Are Acceptances and Orders to be
Yes
No
directed to the above Contact Name
and details?
If ‘NO”, please insert alternative
details
(*)
If an applicant is an individual then the relevant sections identified by (*) will not be
applicable.
Authorisation, Certification and Execution by an applicant (company) (*)
The Director and Director/Secretary named below certify that in submitting this quote on
behalf of the applicant:
(a)
they have read, understood and complied with the requirements of this Invitation;
(b)
the enclosed Response Forms are a true and accurate account of their quote.
Signed for and on behalf of
……………………………………………………………………………….
[insert applicant’s name]
…………………………………………………………….…………..…….
[insert applicant’s ACN]
…………………………………………………………….…………..…….
[insert applicant’s ABN]
in accordance with s.127 of the Corporations Act 2001 (Cth)
this ……………….. day of ……………………..………………. 2…….
by
………………………………………………………..…………………….
[insert full name of Director]
……………………………………………………………………………….
(signature of Director)
………………………………………………………..…………………….
[insert full name of Director/Secretary]
……………………………………………………………………………….
(signature of Director/Secretary)
Where an attorney executes this Deed on behalf of an applicant, the form of execution must
indicate the source of this authority and such authority must be in the form of a Deed and a
certified copy thereof provided to the Agency.
OR
Authorisation, Certification and Execution by an applicant (individual or partnership)
I, the Offer or named below certify that in submitting this quote:
(a)
I have read, understood and complied with the requirements of this Invitation;
(b)
the enclosed Response Forms are a true and accurate account of my quote
Signed for and on behalf of
……………………………………………………………………………….
[insert applicant’s name]
…………………………………………………………….…………..…….
[insert applicant’s ABN]
this ……………….. day of ……………………..………………. 2…….
by
………………………………………………………..…………………….
[insert full name of applicant]
……………………………………………………………………………….
(signature of applicant)
in the presence of
………………………………………………………………………….
[insert full name of witness]
……………………………………………………………………………….
(signature of witness)
(*)
If an applicant is an individual then the relevant sections identified by (*) will not be
applicable.
Response Form 4.2 - Services including Price
The Services offered by the applicant are as follows:
Description
Qty
Unit
Price
(Excl.
GST)
Sub-total
Price
(Excl.
GST)
Price
(GST
compon
ent only)
Total
Price
(Incl.
GST)
List any delivery and installation details and charges
applicable to this quote.
List any taxes, duties or other charges and their details
associated with each Good and/or Service applicable to
this quote.
TOTAL QUOTE PRICE
$
Other Pricing Issues:
Applicants must specify each and every condition that may affect the pricing offered in this
Response Form.
Please specify any other price that may be charged to the Agency for the supply of the
Services under this Invitation.
For each price, specify:
 the nature of the price;
 the circumstances under which it will be incurred; and total price (including GST).
Appendix A: Cognitive Interview Protocols
The proposed method involves four aspects:
 Concurrent observation [CO] – observing the student’s interaction with the items
 Concurrent think aloud [CTA] – using the “think aloud” method on pre-selected items
 Concurrent interview [CI] – a structured question to the student immediately after they
have finished responding to an item
 Retrospective interview [RI] – a series of structured questions about the student’s
experience with the items and the online test overall.
Students are involved in two sessions. Session 1 will include the concurrent observation and
“think aloud” aspects. Session 2 will consist of a retrospective interview.
Generic Protocol Structure
Session 1 (concurrent) – up to 1 hour
Initial Communication to students: Students will be told that the purpose of the sessions is to find
out what skills and knowledge students use in responding to items and whether they find anything
in the presentation or language of items confusing. It is important at this point to explain to
students that their ability in writing is not being assessed and that no judgements about their
academic performance will be made.
The interviewer will explain to students that at certain points they will be asked to think aloud as
they respond to certain items. The interviewer will model what thinking aloud entails with a
sample item presented on paper. Students will also be told that while the interviewer can offer
only limited help and guidance, the students can identify verbally at any time difficulties they
may be having with the items. In addition, the interviewer should make note of any difficulties
the student experiences using the system.
The purpose of the audio-visual recording device will be explained and students will be assured
that the recordings will never be made publically available or provided to any party other than the
contractor or ACARA and that it will only be used for making improvements to the test.
Prior to the students starting the test the interviewer will model to them the “think aloud” method
with a year level appropriate example item. The example will be a NAPLAN-style item of
moderate difficulty appropriate for their year level.
Concurrent Interviewing: As the interview proceeds the interviewer will frame supplementary
questions to ascertain how the student is responding to the writing prompts and to identify the
mental processes that are engaged when writing.
Session 2 (retrospective) – 30 to 40 minutes
Within a reasonable timeframe of sitting the test, students will be interviewed face to face. The
retrospective interview will consists of a set of structured questions including both open-ended
questions and Likert-style responses. The flow of the interview will be from the general to the
specific.
Item specific questions: Students will then be led through a series of questions targeted at specific
issues with specific items. Students will also be asked to identify any particular issues they had
with items that had not been pre-identified.