Advanced Web Testing for Mystery Shopping
Manual Version 1.0
8/2006
Table Of Contents
PRODUCT INFORMATION............................................................................................. 3
ACCESSING TESTCASTER ........................................................................................ 3
QUALIFICATION SETUP AND ADMINISTRATION................................................... 3
QUALIFICATION CATEGORIES AND NAMING CONVENTIONS ....................... 4
LEVELS.......................................................................................................................... 4
ATTACHING TESTS TO QUALIFICATIONS............................................................ 4
GRANTING/REVOKING QUALIFICATIONS MANUALLY ................................... 5
TEST ADMIN – TEST SETTINGS ................................................................................... 7
TEST AVAILABILITY: ................................................................................................ 7
TEST REQUIREMENTS: .............................................................................................. 7
TEST SUBMISSION LIMITS ....................................................................................... 8
JUMP START MODE.................................................................................................... 8
ATTACHING QUALIFICATIONS TO TESTS.......................................................... 10
ADDING QUESTIONS TO TEST............................................................................... 11
UNIQUE QUESTION SETTINGS .............................................................................. 11
SHOPPER TESTING CENTER & TEST TAKING.................................................... 13
CONTINUING A TEST IN PROGRESS..................................................................... 14
TEST REVIEW & GRADING (AUTOMATIC AND MANUAL) ............................. 15
SEARCH FILTERS & THE AUTOSCHEDULER ..................................................... 17
TESTCASTER DEFINITIONS........................................................................................ 19
TESTCASTER CONCEPTS ............................................................................................ 20
FEATURE OVERVIEW .................................................................................................. 21
PRODUCT INFORMATION
Companies have long desired the ability to qualify shoppers with more detail than just a shopper rating and
an MSPA certification. Now there’s TestCaster, an exciting new product from SurfMerchants LLC. that
gives Mystery Shopping companies the ability to evaluate shoppers in their database with company
designed and implemented tests and qualify those shoppers for specific industries, clients, or shop types.
The details of how this module is implemented throughout SASSIE are specified here. If you are more
interested in getting a short step-by-step walk-through of the TestCaster system, we recommend reading
through the “TestCaster Quick Tour” before reading this manual.
ACCESSING TESTCASTER
After the module has been purchased and activated, on the main admin page of SASSiE, under Shopper
Tools, the “TestCaster Test & Quals” option will show. Selecting that option and clicking the Go buttong
will bring up the main administration page of TestCaster.
QUALIFICATION SETUP AND ADMINISTRATION
Three active buttons exist on the main TestCaster Admin Page: View Tests, View Qualifications, and Go
To Test Log.
A “Qualification” is a designation or “merit badge” that a shopper can earn by passing a test (or being
granted one manually by one of your staff). You can create your own qualifications for your company
(these are considered CUSTOM qualifications that only apply within your own system. UNIVERSAL
qualifications apply across all systems, similar to the way MSPA certifications are consistent across all
systems).
Examples of qualifications might be :
* Automotive Approved
* McDonald’s Trained
* Video Certified
* Trusted Veteran Shopper
QUALIFICATION CATEGORIES AND NAMING CONVENTIONS
Qualifications can be grouped into categories for organizational purposes (you might want to create a “Fine
Dining” category if you have multiple Fine Dining qualifications). By default, qualifications will be placed
in the “General” category unless specified otherwise.
For Qualification and Custom Category naming, it is suggested that names be both unique and descriptive
of the use of the created qualification. For instance a general qualification could be as simple as “Activated
Shopper” or be grouped in a category of “Women’s Apparel: Department Stores” or “Karl’s Meat Helmets:
Meat Wearer”.
LEVELS
Levels are used to further detail a qualification in cases where there may be higher or lower versions of the
same qualification.
For example, someone that has shopped Karl’s Meat Helmets (why, we may never know) twice might pass
a test that grants a BASIC (or level 1) qualification, where as someone that has shopped 50 different Karl’s
locations successfully might pass a test that grants an ADVANCED (or level 3) qualification.
NOTE: Levels are always relative to other qualifications in the same category. The higher the level, the
better the qualification.
ATTACHING TESTS TO QUALIFICATIONS
If you want a specific test to grant a qualification when a shopper passes, it must be “attached” to the test
by using the controls at the bottom of the screen.
The drop down displays all tests by name. Selecting the desired one and clicking the add button will
refresh the page with the test name displayed on the qualification form to verify that the selected test is the
correct one.
Once the “Add Qualification” button is clicked, all information on the form is saved and will display on the
Custom Qualifications list. This is also true of clicking the “Save Changes” button on any existing
qualification.
A test can grant more than one qualification. Example: An “Advanced Video Training Test” might grant
the “Basic Video” qualification and the “Advanced Video” qualification.
Also, a qualification might be granted by more than one test. Example: Both the “Mercedes Benz” test and
the “Luxury Car” test might grant the “Luxury Car” qualification.
GRANTING/REVOKING QUALIFICATIONS MANUALLY
On the Custom Qualifications list page, there are links to edit and grant/revoke individual qualifications.
Qualifications can be manually granted at any time completely independent of testing. Some qualifications
like “Trusted Shopper” may have no attached tests at all and may only be granted to shoppers manually by
one of your staff.
Clicking the Grant/Revoke link next to the Qualification name on the list will bring up the Grant/Revoke
page. This page is simply a box with radio buttons for granting or revoking the qualification to or from one
shopper or many shoppers at once (using the shopper search export tool in SASSIE)
Also on the Custom Qualifications page is a total number of shoppers that have been granted that
qualification. Within the shopper logs will be a list of all qualifications a shoppers has earned or been
granted for ease of tracking that information on shoppers.
If listing shoppers out in bulk, those lists will only appear as an online form if fewer than 100 shoppers are
found.
TEST ADMIN – TEST SETTINGS
Clicking the View Tests button will access test administration. If no tests exist the only thing appearing to
administrators will be the header containing the Create New Test button. Clicking on this button will bring
up the test settings screen.
Test names are suggested to be both unique and descriptive of the type of test and/or the qualification
granted by the test.
The email address for the Test administrator at your company must be entered in the EMAIL field. This
person will be responsible for answering any shopper questions regarding this test.
Test status is by default, set to “HOLD” as any “ACTIVE” test will be available to shoppers to take. Once
questions have been assigned to a test, that test can become ACTIVE. The DEACTIVATED option
allows you to “retire” a test (and prevent any further submissions) while keeping all results to that date.
TEST AVAILABILITY:
Controls which shoppers are allowed to take any given test (i.e. available to all shoppers
or only new shoppers)
TEST REQUIREMENTS:
Specifies if new shoppers must take a test, or if they are allowed to skip this test. This
section also sets the parameters of shopper activation: are shoppers activated when they
PASS this test, or just when the TAKE this test.
TEST SUBMISSION LIMITS
Restricts the number of times a test will be taken by shoppers before it will be placed in a “Hold” status.
This setting gives you the ability to limit the number of submissions for this specific test that your company
wants to pay for.
For example, if your goal is to get 1000 shoppers qualified for a specific program but you don’t want to pay
for extra submissions, check this box, enter 1000 submissions, and select “Only count ‘passing’ test
submissions. A current count of submissions is also provided if there have been tests submitted.
Note: Shoppers with submissions in progress will be allowed to complete their submissions even if this
limit is reached.
JUMP START MODE
A “shortcut” option that allows you to select one of the four most popular types of test and have most of the
test settings automatically set for you (individual settings can be adjusted afterwards, if desired). To active
JUMP START MODE, click the word “SHOW”.
Explanations of these presets are as follows.
Measurement Test: Use this type of test to verify how much knowledge shopper has.
Training Test: Use this type of test as a training device, so shoppers can gain knowledge on a specific
industry or client.
Strict Screening: Use this type of test to select the most knowledgeable shoppers for activation.
Loose Screening: Use this type of test to select reasonably knowledgeable shoppers for activation.
Using any of the Jump start mode preset buttons will automatically set Completion, Go Backs, Passes,
Retries, and Scoring to the listed values within the window, however it is possible to adjust values within
the presets for a bit more flexibility.
Without using the Jump Start Mode, Test Administrators can set their own parameters on the test settings.
Completion can be set so :
• There are no special requirements,
• The shopper must continue taking the test until he/she receives a passing grade
• The shopper must continue taking the test until he/she receives a score of 100%
• The shopper is only allowed to pass through the test once.
Go Backs refers to whether or not a shopper is allowed to back up to previous pages while taking a test.
Passes refers to :
• How many times a shoppers is allowed to “go through” all the pages of a test within a single
submission.
• Whether or not the correct answers will appear on subsequent passes.
Retries refers to :
• How many times a shopper can re-take a test if they should fail
• How much time they must wait between those tries.
Scoring controls:
• The predetermined passing grade for the test
• Whether or not the test will be automatically or manually graded
• If the shoppers can see their scores (via web and/or email),
• Whether an email trigger will be used to send and email to the shopper after the test receives a
grade.
Scoring Sections are similar to the sections assigned to surveys in SASSIE. Sectional scores are used to
display to the shopper at the end of the test and display to test administrators on the Grader view.
ATTACHING QUALIFICATIONS TO TESTS
This function accomplishes the same results as attaching tests to qualifications – it may be more convenient
to do it from the test side. Clicking on the drop down will bring up a list of current qualifications that can
be added to the test by clicking the “add” button. All information will not be saved unless the Save
changes button at the bottom of the page is clicked.
The test setting will refresh with a modified header containing five buttons. For an easy way to view the
test settings, click the View Test Details button.
ADDING QUESTIONS TO TEST
In order to add questions to the test click the Edit Test Questions button.
Adding questions to a test is very similar to adding questions to a mystery shopping survey. Here are a few
of the differences and additions:
UNIQUE QUESTION SETTINGS
•
“MUST GET RIGHT” setting: if a question is marked as “Must get right” the shopper will
only receive a passing grade if all “Must get right” questions are answered correctly. When Random
Question Selection is used, Must Get Right questions will always display.
•
“CORRECT” answers : Any answer to a question with a score that is greater than zero is
considered “correct” for purposes of grading.
•
“PAGE
•
RANDOM QUESTION SELECTION is one of TESTCASTER’s strongest tools in preventing
shoppers from cheating by exchanging tests and answers.
BREAK” question type and “RANDOM QUESTION SELECTION”.
By declaring the “number of questions displayed on this page” and adding more questions than the
maximum displayed, TestCaster will automatically select questions at random each time a shopper
takes the test (if a shopper returns to the same page during the taking of the test, Testcaster will
remember which questions were selected previously and show those questions again for
consistency).
•
An option to show a random number of a set of questions is used with the Page Break
Question type.
•
A page can be declared an introductory page, usually containing content to prepare a shopper
for the upcoming test. Introductory pages are not displayed to administrators at the time of
grading. Marking a page of information as introductory is a good way to avoid having to
view it when grading shopper submissions.
The question examples used here are (Radio button – Multi-Choice question type)
“What is approximately15% of $32.59?”
$3.25/$4.51/$4.89/$5.00
“What is the parent company of Taco Bell?”
Kraft Foods/Coca-Cola/ Wal-Mart /PepsiCo
“What is the largest retail store chain?”
Wal-Mart/ Sears /Target /Karl’s Meat Helmets
“What is the parent company of the Scion automobile?”
General Motors / Diamler-Chrysler /Toyota / Honda
“What is the largest Coffee retailer?”
Starbucks/Dunkin-Donuts/Seattle’s Best/Joe’s
“What is the airspeed velocity of an un-laden swallow?”
1-2 knots/2-4 knots/5+ knots /African or European swallow?
Using the “Refresh this page” button on the question list will reload the complete list of all current
questions as well as a page break if one has been set up for the test.
Once questions are complete to satisfaction and the test is ready for shoppers to take, we need to activate it.
Click the “Edit Test Settings” button in the header bar.
ACTIVATING TESTS
Click the drop down next to Test Status - there is now an “active” option (this option does not appear until
the test has questions added). Selecting that option immediately makes that test available for shoppers.
COPY TESTS
There is also the ability to copy any test for purposes of not having to recreate from scratch an entire test
over again. The copied test’s settings and questions will be duplicated and would require a new unique
name.
Cases where this function would be handy is on a deactivated intro test that has been deemed out of date.
Remember that once a shoppers has seen a question, that question can not be deleted, after that point it can
only be deactivated, so creating a copy would give the test admin the ability to make changes to the
questions to have an up-to-date test.
DELETE TESTS
There is also a delete this test button. That button will only remain if there are no submissions for the test.
Once submissions have been received, a test can no longer be deleted.
SHOPPER TESTING CENTER & TEST TAKING
Upon login, all shoppers who have at least one test available to them will see a link to their “Shopper
Testing Center”. The testing center will have a list of available tests and their associated qualifications.
CONTINUING A TEST IN PROGRESS
Once started, a shopper must finish a test and if for a reason the shopper is unable to finish after starting,
the test will remain in a hold status for them to continue at a later time. By clicking “Save” at the bottom of
any test page, the test will go into this hold status and a “Continue Test” link now appears on the “Shopper
Testing Center” page.
Once a shopper has finished answering all required questions of the test, clicking the “Finish” button at the
bottom of the final test page will setup the test for submission to the test administrator. The page will
refresh and a link to “Submit test for grading” will appear. Once submitted, the shopper can only retake a
test if allowed by the test administrator.
The “Start another pass through this test” button will only appear if more passes are allowed in the test
settings or if the shopper must make another pass through the test (because the test requirements are a
passing or perfect score).
Submit test for grading only appears if shoppers are not allowed to take another pass through the test.
TEST REVIEW & GRADING (AUTOMATIC AND MANUAL)
The initial setup of a test determines whether or not a grade for that test will be automatically or manually
granted once a shopper completes a test submission.
If automatically graded, the shopper receives immediate notification regarding score and if passing, the
qualification is immediately granted.
With a Manual grading setup, the test administrator receives notification in the test log that there are a
number of submissions awaiting grades.
On the right hand side of the test log there are links to grade or re-grade all tests, including those tests that
were automatically graded.
On the main admin page of TestCaster, the link for the Test Log will show a total number of submissions
awaiting grades. The Go To Test Log button will bring up the Admin Test Log.
The search tool, works the same way as the logs in SASSiE. Test searches can be filtered by ID,
submission date, shopper names, status, and score. Clicking the GO button will bring up a list of all
searched on submissions.
The existing Pass/Fail shows on those tests that have previously been graded. When a dash ( - ) is shown, it
means the test has not yet been submitted by the shopper and can only be viewed at this time by the grader.
By clicking “Re-Grade Test” the Grader Controls Window will appear. In this window are the controls for
changing the grade and excluding the test if that is deemed necessary by a companies’ test Administrator.
Note: A test submission placed into Excluded status may not be removed from that status.
Graders are unable to change shopper answers on tests, but are able to change the answers to admin-only
questions and the grade the test received if deemed necessary. For instance, if the test questions are found
to be too difficult and an unacceptably large percentage of shoppers are failing, the grades can be changed
in order to scale submissions.
SEARCH FILTERS & THE AUTOSCHEDULER
In place with the TestCaster module is the ability to limit the scope of shopper searches for exports and
bulk emails, by filtering by qualifications, making choosing qualified shoppers for jobs much easier.
The filters give the option of searching for shoppers with any existing qualification as well as asking if
“higher” or “equivalent” levels of the same qualification category should be included in the search results.
This export can also work searching for shoppers that have not been received a specific qualification.
There are new icons next to shopper names when using this filtering method that make it easy to tell at a
glance which of the searched for shoppers is the best qualified.
A red X means the shopper does not have the desired qualification.
A green check mark denotes a shopper with the qualifications selected (or any equivalent or higher levels
specified)
A yellow dash means the shopper does not have any of the selected qualifications, but does have an
UNSPECIFIED equivalent of higher level (so they’re a good shopper to select if no shoppers with green
check marks are available).
These filters are also used in scheduling shops and will show up the same way on a list of shoppers.
TESTCASTER DEFINITIONS
Qualification: A specialized award granted to shoppers that have a certain level of experience in, or
knowledge regarding Mystery Shopping. Qualifications can range from very general to very specific.
Category: A division of Qualifications, to simplify searching and filtering of qualifications. Types of
Categories are by industry, Women’s Apparel, Food Service, or Pet Varnishing, or client specific, Karl’s
Meat Helmets, Mary’s Rocket Surgery, or Harry’s Rabid Sheep.
Level: Levels are used to give a qualification more detail. An example would be, basic, intermediate, and
advanced.
Test: A set of questions and rules that govern those questions, that a shopper may answer to gain a
qualification, or become an active shopper on a Mystery Shopping companies system.
Introductory (Signup) Test: A test presented to new shoppers after signup. Introductory tests may be
required or optional, and may control whether a shopper is activated on a system.
Pass: A complete loop through a test. For tests with pages, a shopper moving from the first to the last page
in a test would be considered a “pass”. For tests with a single page, each time the shopper chooses to
“Finish” the test is considered a pass.
Attempt: Each time a shopper submits a test for grading is considered an attempt.
Score: The overall percentage score a shopper earns based on the points associated with the questions seen
and answers given.
Grade: Whether a test has been passed or failed. May be based on score given automatically or manually
decided by a Grader.
Grader: An administrator reviewing a test submission for the purposes of giving it a grade.
Go Back: Whether the shopper can go backward in a test to view previous pages without having to
complete a pass.
Show Incorrect: Whether a shopper will be shown which of his/her answers were incorrect when returning
to a page after making a complete pass.
Introductory Page: A page that presents a shopper with introductory text and/or images on which
subsequent passes will be based. Introductory pages may not include answerable questions and are for
informational purposes only. Introductory pages are not displayed to Graders.
Must Get Right Question: A question that a shopper must answer correctly in order to pass the test,
regardless of overall score.
Submission: An instance of a shopper taking a test.
Incomplete: A submission that a shopper is in the process of taking. Incomplete submissions are not
editable by Graders.
Completed: A submission that a shopper has filled out in its entirety but has not yet submitted for grading.
Completed submissions are not editable by Graders.
Submitted: A submission that a shopper has completed and submitted for grading, but does not yet have a
grade. Submissions in this status are not accessible to shoppers.
Hold A and B: A submission that a Grader has put aside temporarily. Submissions in these statuses do not
have grades and are not accessible to shoppers.
Graded: A submission that has been given a grade, either automatically or manually by a grader.
Submissions in this status are not accessible to shoppers.
Excluded: A submission that has been discarded. Excluded submissions do not have a grade and may or
may not count against a shopper when it comes to future attempts at the test. Submissions in this status are
not accessible to shoppers. Excluded submissions are not billed for.
TESTCASTER CONCEPTS
Qualifications vs. Tests: Qualifications and Tests do not necessarily have a one-to-one relationship.
Qualifications may be granted manually or by passing one or more Tests. Tests may have no
Qualifications associated with them, or may grant one or many Qualifications.
Score vs. Grade: Any test with scored questions will have scores calculated when a shopper completes a
pass through the test or when a Grader saves a submission in any status other than Excluded. Though a
submission’s passing or failing grade may be based on its score, having a score does not necessarily mean a
submission has a grade. Only submissions in Graded status will have a grade indicating whether the
shopper has passed or failed the test.
Auto vs. Manual Grading: An automatically graded test will give a shopper a grade upon submission of a
test. The submission will then go straight to Graded status without any action on the part of a Grader. A
manually graded test will put shopper submissions into Submitted status and grading will be carried out by
a Grader. Automatically graded tests must have a passing score so that the system can determine whether
the shopper has passed, whereas manually graded tests may or may not have a passing score.
Question Banking: Non-introductory pages may be configured to display a subset of the questions they
contain. This subset is randomly selected for each test submission. In order for random selection to occur,
the number of questions contained on the page must exceed the number to be displayed. It should be noted
that questions will always display in the same order and that Must Get Right questions will always be
displayed.
Question Tracking: Because the questions that a shopper sees may be randomly selected, and questions
may be deactivated after a shopper completes a test, the exact set of questions a shopper has viewed is
tracked for each submission. Each subsequent time a submission is viewed, those questions and only those
questions are re-displayed.
Correct Answers: Only multiple-choice questions may have correct answers. A correct answer is any
answer that has a score associated with it.
Question Deactivation: Because questions are tracked, it is not possible to delete a question that a shopper
has seen. However, questions may be deactivated such that they will not display on any future
submissions. Questions may be reactivated at any time.
Permissions: There are two permissions associated with Shopper Qualifications and Testing. The first,
Qualification and Testing Admin, grants the ability to create tests and qualifications and manually grant
qualifications to shoppers. An administrator must be Form Creation Certified to get this permission. The
second, Qualification Grading, grants the ability to view basic information about tests and grade test
submissions.
Retries: The number of times a shopper may attempt to pass a test may be limited. By default, tests allow
only a single attempt. If a shopper has passed a test, he/she will not be allowed to take it again at any point.
If a test has been failed, the shopper may retake it until he/she passes or exceeds the allowed number of
retries. Excluded test submissions count as failures unless the Grader has opted to make them not count, at
which point they will not be deducted from the number of retries a shopper has remaining.
FEATURE OVERVIEW
Overview of the various pages that make up the TestCaster module.
Testing & Qualification Admin: The main landing page for this module, allows access to the Qualification
and Test lists, as well as the Admin Test Log.
Qualification List & Admin: Displays all qualifications existing for a company, allows them to be modified,
and new qualifications added
Grant/Revoke Qualifications: Allows user to search for individual shoppers or shoppers in bulk and grant
or revoke a specific qualification for those shoppers.
Test List & Admin: Displays all tests existing for an MSP, allows them to be modified, and new tests added
Test Question List & Admin: Displays all questions for a specific test, allows modification of existing
questions, and new questions to be added.
Test Details: Designed for users without permission to modify tests, displays details of a test’s settings and
an un-modifiable list of its questions.
Copy Test: Allows the user to make a copy of an existing test, duplicating all associated questions and
qualifications
View Test: Displays a generic ‘demo’ view of a test without actually creating a stored test submission
Take Sample Test: Allows the user to generate a new test submission and fill it out. Designed to allow a
test to be previewed prior to releasing it to shoppers.
Shopper Login: Has been modified to redirect new shoppers, when necessary, to the Introductory test for
the MSP.
Shopper Log: Displays which qualifications a shopper currently has and gives a link to their Shopper
Testing Center.
Shopper Testing Center: Allows shoppers access to tests they can take and displays a record of tests
previously taken.
Shopper Test View: Allows a shopper to create and fill out a test submission.
Admin Test Log: Displays all existing test submissions for the MSP, filtering and sorting them and allowing
access to the Admin Test View
Admin Test View: Allows a Grader to review and grade test submissions as well as move unwanted
submissions to Excluded
Autoscheduler Profile: Now has an option to filter shoppers by qualifications, both at the application and at
the self-assign level
Autoscheduler Send Emails: If a profile is setup to require qualifications, shoppers who do not have those
qualifications will not receive autoscheduler emails
Shopper Auto Application: Prevents applications by shoppers who do not have required qualifications and
prevents self-assign by shoppers who do not have preferred qualifications.
Process Apps: Displays qualifications relevant to the session’s profile, indicating whether a shopper does or
does not have qualifications.
Shopper Search: Allows searching by shoppers with and without specified qualifications for the purposes
of exporting and emailing.
Shopper Export: Exports all qualifications that a shopper currently has.