PART 1 - INVITATION TO QUOTE
Quotes are invited by the Australian Curriculum, Assessment and Reporting Authority
(ACARA) for a contract in relation to the Trial of the NAPLAN Online Tailored Test Design:
Cognitive interviews – Reading August 2013 (as specified in this document).
Quotes are to be submitted on the basis that they conform to Part 2: Specifications and Part
4: Response forms.
Agency
Australian Curriculum, Assessment and Reporting Authority
Closing Date
5:00pm (Sydney time), Friday 19 April 2013
Contact Officers
For further information you can contact:
Peter Matheson on ph: (02) 8098 3116 or email [email protected]
John Skehan on ph: (02) 8098 3133 or email: [email protected]
Quote Validity Period
Quotes must remain valid for a minimum period of 90 days after the Closing Date.
Lodgement of Quote:
Quotes submitted in response to this Invitation will be accepted by ACARA via Email
You are required to email the full quote as a PDF to [email protected] with a
CC to [email protected] with the following text in the message box Trial of
Tailored Test Design: Cognitive interviews NAPLAN Online – Reading
PART 2 – SPECIFICATIONS
2.1
Introduction
The Australian Curriculum, Assessment and Reporting Authority (ACARA) is responsible for
the development of a rigorous, world-class Australian curriculum from Foundation to Year 12.
To complement the development of an Australian curriculum, ACARA is also responsible for
developing and administering a national assessment program aligned to the national
curriculum that measures students’ progress, and the provision of information, resources,
support and guidance to the teaching profession.
ACARA reports to, and is directed by, the Standing Council on School Education and Early
Childhood (previously known as the Ministerial Council for Education, Early Childhood
Development and Youth Affairs).
ACARA has overall responsibility for implementing the National Assessment Program (NAP).
2.2
Background
National Assessment Program – Literacy and Numeracy
The National Assessment Program — Literacy and Numeracy (NAPLAN) is the main
assessment program conducted by ACARA. Every year all Australian students in Years 3, 5,
7 and 9 are assessed in Reading, Writing, Language Conventions and Numeracy. The tests
are equated longitudinally each year in order to ensure comparability of results from year to
year, and to enable tracking of students’ results.
National Assessment and Surveys Online Program
The National Assessment and Surveys Online Program (NASOP), under the Australian
Government’s Online Diagnostic Tools election commitment, is intended to develop capacity
to deliver national assessments and surveys through an online environment.
This Program comprises three elements:

development, testing and delivery of a national online School Opinion Survey for
deployment in 2013 at jurisdictional discretion (Element 1),

development, testing and delivery online of the 2013 National Assessment Program –
Civics and Citizenship sample assessment (NAP-CC) (Element 2), and

planning and implementing a clearly defined assessment and reporting research agenda
that will allow reporting to SCSEEC on issues and options for delivering NAPLAN online
(Element 3).
NAPLAN Research Study
ACARA has developed a comprehensive research agenda that will provide findings on a
range of issues and provide evidence that will enable SCSEEC to make decisions regarding
the transition of NAPLAN from a pen-and-paper test to a computer-based assessment. The
research study consists of three stages: a pilot research study, a trialling study and a linkage
study.
ACARA completed the first stage, the pilot research study, in December 2012. The second
stage of the research study focuses on trialling the tailored test design.
ACARA has developed a tailored, multi-stage test design for the implementation of NAPLAN
as a computer-delivered assessment. In this tailored test, a student progresses through a
series of testlets (item sets that are constructed as stand-alone mini-tests) based on the
correctness of his or her answers in each testlet. In this way the whole test is tailored to the
ability level of a student. The number of pathways that students can take to complete a full
test is predetermined, based on the number of branching points that determine student
progression through the test and corresponding testlets.
2.3
Purpose
The purpose of this contract is to engage a qualified and experienced Contractor to
investigate the cognitive and behavioural engagement of students with NAPLAN Reading
items delivered within the new tailored (multi-stage) test design.
The proposed tailored test design is a multi-branching test consisting of interlocking item sets
(testlets) covering Years 3 to 9 within each testing domain. In this design each student will
work through three testlets, allocated by the system at two branching points. It will be
possible for each student to take one of five possible pathways. This is illustrated in Figure 1
Figure 1: Illustration of the tailored test design and possible test-pathways
The main benefit of the tailored test design is that it provides better targeting of students
relative to their ability. This is particularly important for assessing underperforming students
and students who have not been sufficiently challenged by existing items. The proposed
design contains a testlet specifically designed to cater for needs of these students at each
year level.
The primary purpose of the cognitive interview will be to investigate the impact of the
multistage-branching design on the students' test taking behaviour. In addition it is expected
that information about students' interaction with key branching testlet/testlets will be
investigated.
At Years 3 and 5 each testlet has 13 items and at Years 7 and 9 each testlet has16 items.
Each student will work through two branching testlets. Key items in the two branching testlets
through which a student is passing should be discussed in detail in the cognitive interviews.
Another key aspect is that the difficulty profile of items in a branched test structure will differ
from the steadily increasing item difficulty which is characteristic of paper-based NAPLAN.
Student responses to items which show marked changes in difficulty from previous items
should be monitored carefully in the cognitive interviews.
Testlets C and F do not have analogues in current NAPLAN tests. Accordingly, the
performance of students who reach these testlets will be of particular interest and should be
examined by the cognitive interviews.
The Contractor will work closely with ACARA to identify key aspects of the cognitive interview
methodology and research plan.
This investigation will be conducted in parallel with the NAPLAN Online Trialling Study
(Reading and Numeracy) utilising the same assessments and a sub-sample of the schools
participating in the trialling study.
The NAPLAN Online Trialling Study will be invigilated by a third party, the NAPLAN Online
Trialling Study Contractor, who will be administering the tests and who will provide assistance
with the test delivery platform.
The testing schedule for the NAPLAN Online Trialling Study will be prepared by the NAPLAN
Online Trialling Study Contractor and provided to the Cognitive Interview Contractors for
identification of schools to participate in the cognitive interviews. The NAPLAN Online
Trialling Study Contractor will coordinate all communications to schools.
2.4
Scope of project
The successful provider will be required to:

Conduct cognitive interviews as one-on-one sessions with a sample of students
participating in the NAPLAN Online Trialling Study. The anticipated sample size is 10
students per test and per year level. At each year level, the first five students will be
allowed to progress through the branching freely. The next five students will work through
a fixed set of items. This will ensure that the same quantity of data is obtained for Testlet
A to D as is obtained for Testlet A to B. Participants should be selected so that at least
two are likely to be directed to testlet C, and at least two are likely to be directed to testlet
F. Where conditions allow and where valid observations could be obtained, two students
could be simultaneously interviewed and/or observed.

Conduct cognitive interviews primarily in metropolitan schools but also in some regional
schools. It is further expected that the students will be recruited from at least three
different schools at each year level.

Liaise with ACARA in the construction of the sample of schools and/or students that will
be included in this study. The schools will be sampled from those participating in the
NAPLAN Online Trialling Study.

Participate in training on the online test delivery system which will be provided by
ACARA’s Test Delivery System Contractor.

Work with ACARA to finalise protocols for conducting the cognitive interviews, drawing on
existing protocols developed for ACARA in previous studies (see Appendix A). Methods
utilised in previous studies involved four aspects:
o
Concurrent observation– observing the student’s interaction with the items
o
Concurrent think aloud – using the “think aloud” method on pre-selected small
number of items and/or test-pathway junctions
o
Concurrent interview – a structured question to the student immediately after they
have finished responding to an item
o
Retrospective interview– a series of structured questions about the student’s
experience with the testlets, branching and the computer delivered test overall.
Students will be involved in two sessions. Session 1 will include the concurrent
observation, concurrent interview and “think aloud” aspects. Session 2 will consist of the
retrospective interview.

Observations/interviews are to be conducted by interviewers who have relevant
experience in cognitive interviewing as well as an understanding of the construct being
tested. The Contractor will provide ACARA with evidence of the interviewers’ experience
and qualifications.

It is expected that the Contractor will provide training specific to this project to ensure
interviews are conducted consistently by the interviewers. A combined video recording of
the screen and audio recording of each student is to be made for all sessions. Once all
interviews have been completed, the records should be collated into spreadsheets and
combined with the student response data from the online platform.

If electronically recording the interview the Contractor will be required to obtain consent to
interview from the parents/guardians of students that will be included in the study, and
only use the information from the cognitive interviews for the purpose of the study.
ACARA will provide a pro forma consent agreement to be used by the Contractor.

The Contractor will liaise and coordinate with the:

2.5
-
Test Delivery System Contractor regarding the test delivery system
-
NAPLAN Online Trialling Study Contractor and the Numeracy Cognitive Interviews
Contractor regarding the times and dates of cognitive interview implementation
and communication to schools
-
Test administrators for the NAPLAN Online Trialling Study on the day of the tests
regarding the specific students who will be participating in the cognitive interviews.
Produce a report summarising the main findings of the cognitive study, particularly
regarding the engagement of students with the tailored test design. The Contractor is
required to obtain an accurate record of each interview either through electronic recording
or by transcribing the interview.
Timing
Note: The timeframe will be confirmed during contract negotiations, and the final
timeframe will be included in the Contract. Organisations should use the
following information as the basis for preparing their responses. Although
ACARA is prepared to discuss changes to the proposed timeframe, it reserves
the right not to alter the proposed timeframe.
It is expected that this project will commence in June 2013 and be completed by midNovember 2013.
Task name
Start
End
Training on Test Delivery System
June 2013
June 2013
Finalisation of school sample for cognitive
interviews
19-Jul-13
19-Jul-13
Finalisation of cognitive interview protocols and
scripts
26-Jul-13
26-Jul-13
Implementation of the cognitive interviews
12-Aug-13
6-Sep-13
Draft Report on cognitive interviews
27-Sept-13
7-Oct-13
Final Report on cognitive interviews
8-Oct-13
6-Nov-13
PART 3 – EVALUATION PROCESS & CRITERIA
3.1
Evaluation Process
The evaluation process will involve an assessment of quotes received against the criteria
listed below. ACARA reserves the right to short-list during the evaluation process using the
evaluation criteria contained below. The evaluation process may also involve discussions,
reference, financial and corporate checks.
3.2
Evaluation Criteria
Offers received will be evaluated against the following criteria (if applicable). The weightings
ascribed to each criterion will be applied consistently.

Ability to meet the Objectives/Scope outlined in the Specification

Reputation and Experience (track record, competence, capacity to deliver)

Ability to meet deadlines and conduct activities in a timely fashion

Value for money.
PART 4 – RESPONSE FORM
(The Applicant must complete all of the following Response Forms as part of its
Quote)
Response Form 4.1 - Applicant Authorisation and Certification
Information Required
Name of applicant:
 if applicant is a company –
specify the full company
name;
 if applicant is a trust – specify
the names of each trustee of
the trust;
 if applicant trades under a
business name, specify the
registered business name; or
 if applicant is an individual or
partnership, specify the full
name of each individual or
partner.
Applicant’s ACN (if applicable)
Applicant’s ABN
State or Territory in which Business
/ Corporation is registered (*)
Name of Holding Company /
Corporate Group (*)
Company Profile
Details
OR if no ABN – have you submitted a ‘Statement by
a supplier – Reason for not quoting an Australian
Business Number (ABN) to an enterprise’ form?
Yes
No
As part of the quote process, each applicant must
include a copy of their Company’s Profile.
Does your quote include your Company’s Profile:
Yes
No
Yes
No
Is the applicant registered for GST?
Postal Address
Street Address
Contact Name
Contact Phone
Contact E-mail
Contact Fax
Is it proposed to sub-contract any
Yes
No
part of the Goods and/or Services?
If “YES”, please specify full name
and address of each sub-contractor
and their relevant experience and
expertise in relation to the offered
Goods and/or Services.
Are Acceptances and Orders to be
Yes
No
directed to the above Contact Name
and details?
If ‘NO”, please insert alternative
details
(*)
If an applicant is an individual then the relevant sections identified by (*) will not be
applicable.
Authorisation, Certification and Execution by an applicant (company) (*)
The Director and Director/Secretary named below certify that in submitting this quote on
behalf of the applicant:
(a)
they have read, understood and complied with the requirements of this Invitation;
(b)
the enclosed Response Forms are a true and accurate account of their quote.
Signed for and on behalf of
……………………………………………………………………………….
[insert applicant’s name]
…………………………………………………………….…………..…….
[insert applicant’s ACN]
…………………………………………………………….…………..…….
[insert applicant’s ABN]
in accordance with s.127 of the Corporations Act 2001 (Cth)
this ……………….. day of ……………………..………………. 2…….
by
………………………………………………………..…………………….
[insert full name of Director]
……………………………………………………………………………….
(signature of Director)
………………………………………………………..…………………….
[insert full name of Director/Secretary]
……………………………………………………………………………….
(signature of Director/Secretary)
Where an attorney executes this Deed on behalf of an applicant, the form of execution must
indicate the source of this authority and such authority must be in the form of a Deed and a
certified copy thereof provided to the Agency.
OR
Authorisation, Certification and Execution by an applicant (individual or partnership)
I, the Offer or named below certify that in submitting this quote:
(a)
I have read, understood and complied with the requirements of this Invitation;
(b)
the enclosed Response Forms are a true and accurate account of my quote
Signed for and on behalf of
……………………………………………………………………………….
[insert applicant’s name]
…………………………………………………………….…………..…….
[insert applicant’s ABN]
this ……………….. day of ……………………..………………. 2…….
by
………………………………………………………..…………………….
[insert full name of applicant]
……………………………………………………………………………….
(signature of applicant)
in the presence of
………………………………………………………………………….
[insert full name of witness]
……………………………………………………………………………….
(signature of witness)
(*)
If an applicant is an individual then the relevant sections identified by (*) will not be
applicable.
Response Form 4.2 - Services including Price
The Services offered by the applicant are as follows:
Description
Qty
Unit
Price
(Excl.
GST)
Sub-total
Price
(Excl.
GST)
Price
(GST
compon
ent only)
Total
Price
(Incl.
GST)
List any delivery and installation details and charges
applicable to this quote.
List any taxes, duties or other charges and their details
associated with each Good and/or Service applicable to
this quote.
TOTAL QUOTE PRICE
$
Other Pricing Issues:
Applicants must specify each and every condition that may affect the pricing offered in this
Response Form.
Please specify any other price that may be charged to the Agency for the supply of the
Services under this Invitation.
For each price, specify:
 the nature of the price;
 the circumstances under which it will be incurred; and total price (including GST).
Appendix A: Cognitive Interview Protocols
The proposed method involves four aspects:
 Concurrent observation [CO] – observing the student’s interaction with the items
 Concurrent think aloud [CTA] – using the “think aloud” method on pre-selected items
 Concurrent interview [CI] – a structured question to the student immediately after they
have finished responding to an item
 Retrospective interview [RI] – a series of structured questions about the student’s
experience with the items and the online test overall.
Students are involved in two sessions. Session 1 will include the concurrent observation and
“think aloud” aspects. Session 2 will consist of a retrospective interview.
Generic Protocol Structure
Session 1 (concurrent) – up to 1 hour
Initial Communication to students: Students will be told that the purpose of the sessions is to find
out what skills and knowledge students use in responding to items and whether they find anything
in the presentation or language of items confusing. It is important at this point to explain to
students that their ability in reading is not being assessed and that no judgements about their
academic performance will be made.
The interviewer will explain to students that at certain points they will be asked to think aloud as
they respond to certain items. The interviewer will model what thinking aloud entails with a
sample item presented on paper. Students will also be told that while the interviewer can offer
only limited help and guidance, the students can identify verbally at any time difficulties they
may be having with the items. In addition, the interviewer should make note of any difficulties
the student experiences using the system.
The purpose of the audio-visual recording device will be explained and students will be assured
that the recordings will never be made publically available or provided to any party other than the
contractor or ACARA, and that it will only be used for making improvements to the test.
Prior to the students starting the test the interviewer will model to them the “think aloud” method
with a year level appropriate example item. The example will be a NAPLAN-style reading item
of moderate difficulty appropriate for their year level.
Concurrent Interviewing: When a student reaches one of the pre-identified items, the student will
be interrupted and asked to think aloud as they respond to the item. After they respond to some
items, the interviewer will ask students how they went about solving the question. Students will
be interviewed on no more than 10 questions at this stage.
Session 2 (retrospective) – up to 1 hour
Within a reasonable timeframe of sitting the test, students will be interviewed face to face. The
retrospective interview will consists of a set of structured questions including both open-ended
questions and Likert-style responses. The flow of the interview will be from the general to the
specific.
Item and testlet specific questions: Students will be led through a series of questions targeted at
specific issues with specific items and or testlets. Students will also be asked to identify any
particular issues they had with items that had not been pre-identified.