EVERNOTE
Usability Test Kit
Kimberly Ha | Keane Riley | Noah Eisfelder | Bonnie Tran
HCDE 417
November 11th, 2016
Description
Digital note-taking applications such as Evernote can be major productivity tools for students and
professionals. Supported by several platforms (mobile, desktop, web), Evernote offers functionality
such as creating notebooks, clipping web articles, and notifications. We plan to evaluate Evernote’s
reminder feature as used within the web application.
Contents
Pre-screener Background Questionnair​e
2
Introductory Script
4
Study Script
5
Consent Form
7
Pre-test Questionnaire
8
Task Scenarios
9
Post-task Questionnaire
10
Post-test Questionnaire / Interview
11
Data-logging / Note-taking Forms
12
1
Pre-screener Background Questionnaire
[Note: This questionnaire will be distributed as an online survey using Google forms]
We are a team of students at the University of Washington seeking participants for a usability study.
We are evaluating features in a digital note-taking application. If you are interested, please fill out the
survey below.
Please keep in mind that this study is for educational purposes only. Participation is voluntary. The
study will take place on University of Washington Seattle campus and will require a commitment of
up to 45 minutes. The test session will be video and audio recorded. All participant information will
remain confidential. Participants must be currently taking classes for this quarter, Autumn 2016.
1. Please select your status
❏ Undergraduate
❏ Graduate
❏ Other _______
2. Which of the follow notetaking applications have you used to take class notes?
❏ OneNote
❏ Evernote
❏ Apple Notes (Mac)
❏ Microsoft Word
❏ Google Docs
❏ I do not take notes digitally (handwritten)
❏ Other ______
3. Have you used Evernote before?
❏ Yes
❏ No
If “Yes” to Q3, answer the following questions Q4 and Q5:
4. How long did you use Evernote for?
❏ Tried it out
❏ For a few days
❏ At least one week
❏ At least a month
❏ Still using it consistently
5. Please select the features that you have used and/or are familiar with.
❏ Creating a note
❏ Creating a notebook
❏ Share notes
❏ Tag notes
2
❏ Reminders & notifications
❏ Clip web content (images, PDFs, etc)
❏ Other: ________________________
6. Please provide us with your name and contact information. We will contact you via this email
address if you are selected to participate in our study.
❏ Name: _______________________________
❏ Email: _______________________________
❏ Phone (optional): _______________________
7. Please indicate your availability among these dates and times
Tuesday, 11/15
❏ 9:00am
❏ 10:00am
❏ 11:00am
❏ 12:00am
❏ 1:00pm
❏ 2:00pm
Wednesday, 11/16
❏ 9:00am
❏ 10:00am
❏ 11:00am
❏ 12:00am
❏ 1:00pm
❏ 2:00pm
Thursday, 11/17
❏ 9:00am
❏ 10:00am
❏ 11:00am
❏ 12:00am
❏ 1:00pm
❏ 2:00pm
If you meet our criteria, we will contact you within the next week to provide you with more details.
Thanks for taking the time to complete our survey! If you have questions, contact us at
[email protected].
3
Introductory Script
Thank you for taking time out of your day to participate in this usability study. My name is _____
and I will be guiding you through today’s session. Here with me today is ______ who will be
making sure everything runs smoothly and taking some notes during the session. Before we
begin, I am going to read this script so everyone in the room receives the same instructions and
we are all on the same page.
The reason you are here today is because we are asking some people to try out some features
on Evernote and see whether or not they work according to how ​you​ expect them to work. This
is important because we are ​not​ testing you, we are testing the application. You cannot do
anything wrong during this study. Our objective is to improve Evernote. We did not design or
build Evernote so you should feel free to give your honest opinions and thoughts during the
session.
For the purpose of our study, we are going to use a webcam, audio recorder, and
screen-recorder so we can review the session later. The video will only be used to help us
improve the application and will not be seen by anyone except the people working on this
project and for reporting purposes. We have a consent form for you to sign today. Do you have
any questions so far before I explain what you will be doing today?
Here is the consent form for you to sign
❏ Give participant consent form to sign
Today, you will be given a packet with four tasks to complete on your own. It will take
approximately 30 minutes.
I want you to:
● Read each scenario out loud and use the computer in front of you to complete the task
● After each task, remember to complete the rating on how easy the task was on the page
● After all the tasks have been completed, we will have an open discussion about your
experience using Evernote and we’ll ask any clarifying questions based on our
observations and notes at the end
While completing each task, I want you to verbally think aloud—tell us what you see on the
screen, what you’re doing, and your thought process so we can follow along. Any opinions, what
you like, what you dislike, things you find confusing—just throw them out there. If you have any
questions during the study, feel free to ask them. If you need a break at any point, just let us
know.
Do you have any questions before we begin?
4
Study Script
❏
❏
❏
❏
❏
Morae will be configured on the computer to begin
Initial state of Evernote set
Welcome participant, have them seated in front of computer with screen off/black
Ask ​Pre-test ​questions
Read the​ Introductory Script
Here is the packet of tasks. Read each scenario out loud and then begin. Reminder to think aloud
through your process.
❏ Hand participant stapled packet of tasks according to ​table​ below
Participant
Task Order
Pilot
1, 2, 3
P1
1, 2, 3
P2
1, 3, 2
P3
1, 2, 3
P4
1, 3, 2
P5
1, 2, 3
P6
1, 3, 2
P7
1, 2, 3
P8
1, 3, 2
❏
❏
❏
❏
❏
Allow user to work through each task
Remind user to think aloud if needed
Remind users to fill out ​post-task ​ratings and question
Mark the timestamp when they finish all the tasks before open discussion
Stop Morae
Thank you for completing all the tasks. Before we end, I want to ask you some questions about your
general experience using Evernote and its reminder feature.
❏ Ask ​Post-test questions
❏ Ask observers if they have any follow-up or clarification questions for user
We are finished now. Do you have any further questions?
❏ Stop audio recording
❏ Thank the participant and escort them out
5
Consent Form
I agree to participate in the usability study testing Evernote, conducted by a student group at the
University of Washington, Human Centered Design and Engineering Department. During this study:
●
●
●
●
I will be given a hypothetical scenario and asked to perform tasks on a computer while
dictating my thought process
I will be interviewed about tasks I performed during the session
I will be asked for relevant experience and demographic information
I will be recorded by audio and video during the session
I understand and consent to recordings being used and released by the Evernote Test Group at the
University of Washington, Human Centered Design and Engineering Department. I understand that
the information and recording is for research purposes only and that my name and image will not be
used for any other purpose. My name and any other identification will remain confidential when used
to describe the findings, and will not be attached to recordings or any other information.
I relinquish any rights to the recording used by the Evernote Usability Test Group at the University of
Washington, Human Centered Design and Engineering Department without further permission. I am
participating in this study voluntarily. The descriptions and findings may be used to suggest
improvements to the Evernote website design. I can withdraw consent to the experiment and stop
participating at any point during the session.
Below is my signature indicating that I have read and understood the information on this form and
that any questions or confusion I might have about the session have been answered.
Date: _________
Participant’s printed name: __________________________________________________
Participant’s signature : ____________________________________________________
6
Pre-test Questionnaire
Please briefly introduce yourself and the classes you are taking this quarter.
What productivity/organizational note-taking tools or systems do you currently use?
In what context(s) do you use them in?
What do you like about the tool(s) you currently use?
How do you feel about setting alarms/notifications? In what situations do you see
yourself using these in?
7
Task Scenarios
Task # 1
Goals
Attach a reminder to an existing note
Scenario
Your PSYCH 101 professor is giving you the
questions you need to know for your next
“Homework 4” assignment. You are taking notes on
these assignment details. She then mentions that it
is due on the Wednesday three weeks from today.
You want have Evernote to keep track of this date
for you.
Output
‘Homework #4’ note has an attached reminder for
the proper date
Task # 2
Goals
Create a reminder with a specific date for a quiz
Scenario
Your PSYCH 101 professor just announced that
your class has “Quiz #4” next Friday on the 25th!
You gotta keep track of this date somehow so that
you don’t completely forget about it.
Output
A reminder with the proper date is added. At this
point, user will have entered the “Notify Me” option
Task # 2.1
Goals
Edit the reminder’s name and date
Scenario
Seems like your PSYCH 101 peers are struggling
with the class concepts. Your professor decided to
push back the date of the quiz to Dec. 2nd. She
also changed the name of the quiz to “Quiz #5”.
Update your notes to account for these changes.
Output
The reminder has the corrected date
Task # 2.2
Goals
Remove reminder and note attached
Scenario
“Quiz #5” on Wednesday has been canceled! Hip
hip hooray! Yip yip yoohay! Better get rid of your
reminder so that you don’t freak yourself out
Tuesday night.
Output
The reminder and note for the quiz is deleted
8
Task # 3
Goals
Attach a reminder to an existing note
Scenario
During PSYCH 101 lecture, your professor
announces that there is a midterm review session
next Friday at 9:30pm. Set the day and time for you
to study these notes next week!
Output
A reminder is attached to ‘Midterm Review Notes’
Task # 3.1
Goals
Remove the reminder but keep the note
Scenario
Midterms are over! You don’t need to review your
midterm notes anymore. Update your Evernote as
you see appropriate.
Output
User has either crossed off or removed the
reminder from ‘Midterm Review Notes’. The note
itself still exists
Post-task Questionnaire
After each question, participants will be asked to answer two brief questions written on the paper
underneath each task:
1. This task was:
​Very straightforward​
1
2
3
4
5
​Very confusing
2. If you were confused by any part of this task, please briefly describe your confusion:
These responses will be compared with the post-test questionnaire responses to see if participants’
ratings of individual tasks line up with their rating the whole reminder system. This will also identify
pain points that we can focus on providing a design solution.
9
Post-test Questionnaire / Interview
What was the most difficult part of using the Reminder feature? Why?
How would you describe the reminder functionality? What would it be used for?
Rate the usefulness of the Evernote reminder feature
Not useful
1
2
3
4
5
Very useful
Do you prefer to create a reminder from ​within the note​ or ​from the sidebar above the
note list​, and why?
Are there any additional comments/thoughts/feedback you’d like to share?
If you have any ideas for improvement, feel free to share them.
10
Data-logging / Note-taking Forms
Task Description
# steps
Task
completed?
Number of
errors
Number
of hints
Attach reminder to an existing note
Create a reminder for Quiz #4
Edit a reminder for Quiz #4
Remove the reminder along with the
attached note
Attach reminder to ‘midterm review notes’
Clear the reminder only
Task Description
Observations of participant behavior
Attach reminder to an
existing note
Create a reminder for Quiz
#4
Edit a reminder for Quiz
#4
Remove the reminder
along with the attached
note
Attach reminder to
‘midterm review notes’
Clear the reminder only
11