With the support of the Lifelong Learning Programme of the European Union
AGREEMENT NUMBER: UK/12/LLP-LdV/TOI-584
PROJECT NUMBER: 2012-1-GB2-LEO05-07950
Refugee Interactive Skills for Employment
Leonardo da Vinci – Transfer of Innovation
Title
Consolidated Usability Report
Deliverable No.
9
Deliverable Title Usability Evaluation Report
Work Package
5
Work Package
Usability Evaluation
No.
Title
Status (D: draft; RD: revised draft; F:
D
final)
Authors
GHI
File Name
RISE Consolidated Usability Report.doc
This project has been funded with support from the European Commission.
This publication reflects the views only of the authors,
and the Commission cannot be held responsible for any use
which may be made of the information contained therein.
Version History Table
Version no.
1
Date
24/07/2014
Author(s)
GHI (Helen Walker)
Comments
Initial Draft
Consolidated Usability Report
A period of usability testing ran between April and July with all iterations of the four games being
tested by partners and by stakeholders including representatives of refugee support organisations,
refugees and employers. It was emphasised to external testers that these were prototypes of the
games and that their feedback and recommendations for improvements were of vital importance for
the further development of the resources.
All partners tested the games for functionality, usability, relevance and engagement and reported
back to the games developers at NTU. Functional evaluation forms were created to record feedback
and issues raised were also reported to the software engineers via the “new issue” tab on the
project management tool, Redmine. This proved to be an extremely useful and effective way of
reporting bugs, concerns, and technical problems etc. as the games developers were notified about
new issues immediately and very often they responded immediately with advice and fixes.
Partners were informed by the software engineers via email each time a new version of the games
was ready for testing and were directed to the password protected section of the project website
where the games were available for download. Instructions were helpfully given on how to install
the games and if partners encountered any problems with downloading or installing the games, the
software engineers offered support via Skype and email to ensure successful installation.
Regular partner meetings were held throughout the usability testing period via Skype where
partners could share their experiences with testing and help and advice was given.
As the usability testing phase moves into the more formal piloting phase, all partners agree that the
rigorous testing both in house and by external stakeholders has resulted in four technically robust,
engaging, relevant, usable and useful interactive resources that we are all confident in and are
excited to take out to our various piloting sites for final improvements, adjustments and refinement.
This report brings the results of each partner’s usability testing phase together, showing how issues
encountered during testing were reported and addressed, a process that has resulted in the current
(but not final) versions of the games with which we are all piloting throughout July, August and
September.
Game 1 Risky Business
Risky Business is a desk top game with a 3d simulated environment of a factory, where the player
has to negotiate their first day at work, learning about health and safety as they tour around the
factory. During this tour of the factory, the player must complete various tasks, collect relevant
safety equipment, identify health and safety signs and recognise hazards. The game ends with a
health and safety quiz in which the player has to score a certain amount to pass in order to complete
the game.
Consultation with employers means that both the environment and the content for this game are
accurate and realistic.
Partner feedback:
U o E:






the game has become more and more user friendly. There are more indication about what
need to be done in order to go on in the game;
Very useful has been to highlight in green what need to be “touched” to go further;
The graphic in the last version of the game (3.0.0) is fantastic, but the game has become
very slow to play;
The movement of the character in the environment has been improved but can still create
some problems by touring right and left;
The player needs to have pretty confident in how to use a Pc (i.e. not usable by person
without ITC knowledge);
The learning contents in the game are very useful;
The complex structure could cause some lack of interest in playing the game till the end;
Some aspects / performance need to be reviewed;
Indication as the map, the mission have been good improved;
Install procedure has become easier

















Initial graphics need improving
Couldn’t view objects
Couldn’t access map
Mission not explained clearly
See no need for 1st person view – too confusing
Too difficult to move around
Due to unfinished state of the game you can’t do a lot
Need more female characters
Feedback needs to be more comprehensive
Some signs in factory don’t glow to indicate they are usable
It’s not obvious that you have to scroll through text to see all of answers
Some of the signs are fuzzy
Being able to move the character up and down is unnecessary and disorientating
Can walk through walls
Not always clear who is talking
The text is too small in the conversation to be read easily
Remove Irish as a language option (people will expect Gaelic)




TIC:


GHI:





















Some of the language used is too complex for the target group.
Door swings through character’s body
Extremely difficult to move the character around- very frustrating for people testing!
Environment is confusing and basic – all looks the same
Instructions aren’t very clear
Not always clear what we have to do
Would be better to have all answer options visible so easier to select
It’s easy to accidentally select the first option by pressing space bar
Characters’ movement is terrible – swaying and moonwalking
Have the read/talk/open box as a clickable button instead of having to press space bar
Sometimes the mission bar doesn’t reflect accurately what you have to do – says meet
Tomasz when you have to clock on
Clocking on should show animation of how to do it
Receptionist tells you the office is 3rd door on the left, but it is 3rd on the right
Tomasz says here is a plan of the building, but doesn’t give you anything
The door to the office goes through the character over and over again
Characters seem to dance or walk on the spot
We got hit by an invisible forklift truck from behind – no way of knowing it was there
No one is driving the forklift
Not clear how to identify the hazards
Couldn’t get past the hazard stage as kept getting taken to hospital
Some minor text errors – someone has forgot to close the gate
Tomasz stands too close to the PPE poster which causes the read/talk buttons to be overlaid
The characters don’t always face each other during conversations
It is clear from these points that partners experienced and reported many of the same faults, bugs
and issues. Many of these have been addressed by the software developers as can be seen in the
vastly improved piloting version of the game that is currently being used.
The issues tracker on Redmine shows the status of reported issues, whether they are in progress or
resolved.
It was generally agreed by testers that when finished, this game will be an enjoyable and useful
resource.
Game 2 Getting There
Getting There is a mobile based game (also playable on desk top) that consists of a simple, stylised
road along which the player directs a car. By tapping the left and right corners of the screen (arrow
keys on desk top) the player moves the car to avoid pothole and to bump into a green cross. Upon
reaching the green cross, a question is triggered with a choice of three answers. If the player choses
the correct answer, their health bar at the top of the screen will reflect this; if the answer is incorrect
one life is lost. The player has three extra lives. There are three levels to the game; the first offers up
questions relating to rights and responsibilities at work; level two deals with services and orientation
and the third relates to cultural differences in the work place. There is an easy to use editor that
enables updating/personalisation of the quiz questions.
Partner feedback:
U o E:





The short version of the game is much better but still too long;
It would be necessary to give more information about how to play;
Feedback about the given answers (“right” and if not showing the correct one);
The game could give/have more resources (i.e. link to external web-pages);
Easy to install.











Easy to download & install
Easy to run game
Needs better instruction
Easy to move character
Questions where easily readable
Differing sizes of text where a bit annoying but seemed necessary
Easy to pause & resume game
Hard to understand feedback at first due to lack of instruction
Flickering bar at top/ bottom 3rd of screen when moving car from side to side
Screen resets after questions (not the same image with potholes…)
‘few feet’ phrasing when asking how far away to stand when talking to another person ->
better: arm’s length
Question on the employer firing you is not challenging
Questions repeat
Endless 3rd level -> cannot win
You lose screen: your score more appropriate?
Have a score board to compare between your scores
More driving would keep my attention better
TIC:






GHI:











It seems odd to have to deliberately hit the green cross to trigger the questions – could they
be barriers that cause the car to stop instead?
The scoring system seems bizarre – scored 48million!
Not obvious if you get the answer right
The levels are way too long – should be maybe 10 questions per level
Shouldn’t get the questions repeating over and over again especially if you got it right
Instead of having the three levels being different categories, have a random selection from
all three in each level – just have level 2 faster that level 1 and level 3 faster than level 2 –
this would make the game more interesting
The font is far too small when answer options are long
If you edit the questions, maybe to be more specific to the local area, will that affect the
game being downloaded by other people in other locations?
A simpler scoring system and feedback screen – you got 10 right well done – would be
better
You lose! – this is a bit harsh!
Need to have links to sites where the information needed to answer the questions can be
found
Testers found this game initially very engaging and easy to play. The main criticism was of the length
of the levels as people found it frustrating to be presented with the same question many times.
Trainers who tested it felt that the ability to edit the questions to reflect specific learning was
extremely useful. The simple colourful graphics of this game were felt to be a real positive element
and certainly were an engaging factor.
A large percentage of the issues raised throughout the testing period have been addressed by the
games developer resulting in a piloting version that is immediately playable and engaging.
Game 3 Interview Expert
Interview Expert is a 3d simulated environment desk top game. The player choses either a male or
female character and is then presented with a waiting room in which there are various posters
containing links to helpful websites relating to workers’ rights and responsibilities, health and safety,
public transport and job searching. Once the posters have all been read, the player goes into an
interview room where an interview for a job takes place. By selecting the correct answers to the
interview questions, the player’s confidence levels increases. At the end of the interview a feedback
screen gives a score and useful advice.
Partner feedback:
U o E:





The movement of the character in the environment has been improved but can still create
some problems by turning right and left;
The player needs to have pretty confident in how to use a Pc (i.e. not usable by person
without ITC knowledge);
How to read the poster and to get their info has been improved;
Good learning contents;
Good idea of inserting the audio.
TIC:












It is not apparent that you can find the language option in the settings because the question
about language is asked and then it brings you to settings, not straight to the language
option.
My left and right controls didn’t seem to be working so this was problematic
Maybe instruction/info – ‘if you click on this an internet browser will open’ should appear.
Difficult to position yourself correctly to know where it will become sensitive to the space
bar
Couldn’t minimise the screen to be able to fully see the answer option and to choose, they
were obscured by the bar at bottom (my computer issue no doubt)
The game begins with the interviewee facing the door. It’s confusing because it might seem
that he/she is outside of the interview room.
In the settings part –Risky Business is specified
Language should be changed in interview dialogue – remove ‘wow’ (interviewer response to
mention of racist incident by interviewee)
Question posed to interviewee about how to greet the interviewers - ‘Do you say ‘– should
have ? after it.
It might not be obvious to the player that they have to hit the space bar to make the phone
stop ringing and move on (that they are responsible)
There is no indication which interviewer you need to stand by
remove the up and down movements, unless this is required to pick something off the floor.
It can be disorienting.





Player should begin the game by sitting down in the waiting room
Some of the interview questions could be more realistic.
Opened multiple (health and safety posters) links, had to restart the game
Hard to move around
Need to go and greet the interviewer but there is no handshake
GHI:

















The controls are too confusing – hard to move character around
Multiple web pages opened up when we clicked on posters
Disembodied grin appeared on one of the posters!
The avatars are scary – especially Zeya and Deidre
The swaying of the characters is off putting
The pint glasses on the table are inappropriate
The door to the interview room opens through the character
Should have the interviewers asking for ID (include a bill with address on)
Player has to get really close to trigger interview conversation
The text box for answers should be bigger so all of the answer is visible
The settings screen has wrong details in
We walked backwards into the interview room – shouldn’t be able to do this
The confidence scale levels need to be simpler –start at terrified and move up to confident
There is some text localised to Ireland still in the English game
German displayed next to confidence scale at start
Some audio is missing – why is there a gap in your cv
Should show them shaking hands and sitting down
Again, many of the same issues and bugs were encountered by all partners. Throughout the usability
testing period when these issues were reported to the software engineers they were quickly acted
upon and either resolved or an explanation given as to why something couldn’t be sorted out. The
addition of audio was received very positively by testers who felt that it added an extra dimension to
the game and made it instantly more accessible to the target group.
The current version being used for piloting has had many of the above issues resolved, although
some people still find the character difficult to control. Feedback has been extremely positive with
trainers and support organisations reporting that they would definitely incorporate this game into
their sessions on interview skills.
Game 4 Email Know-How
Email Know-How is a game developed for mobile devices (tablets, smartphones etc.) Potentially it
will also be made available for desk top use as well. It consist of a mock-up email interface where the
player has to decide what is the best way to respond to various emails, whether to use formal or
informal language. A number of emails are received by the player, all requiring different levels of
formality in their responses. Some spam emails are also received which encourages the player to be
aware of emails requesting passwords etc. When responding to the emails, the player is offered a
choice of possible responses and they accumulate a score depending on how appropriate their
selection is.
Partner feedback:
U o E:

Useful to see and eventually change the chosen answers by clicking (and not to start from the
very begin every time);
Useful the quick and short feedbacks;
The game gives a good idea of formal / informal language;
Easy to install.



TIC:





Need to figure out to hit the ‘send button’ after hitting ‘ready to send’
Need to explain that the home button lets you access new emails
Scrolling down to access new emails is difficult as emails keep opening up when you scroll
No feedback on trashing spam emails
“p” missing from some emails
GHI:









Can’t scroll down to read all of feedback
Can’t scroll down to access new emails as previous emails keep opening instead
It’s hard to make an informed decision on what text to chose if you don’t know what comes
after it
Why do you have to select text when there isn’t a choice? (Just offered “Tom” but still have
to click on it)
No consistency in the pass level needed to move on – sometimes scored full marks but was
told to try again, sometimes scored 75/115 and could move on
Some of the options are a bit subjective – why is single handedly right rather than solo?
Would be good to know how many emails there are so you know where you are in the game
Have the name entry automatically capitalise the first letter
If playing in landscape, there isn’t enough of the screen visible to be able to see the button
you need to click on to move on
The apparent simplicity of this game really appealed to our testers. The fact that it looked like a
real email programme meant that they could immediately see the potential and felt that it was a
professional resource that offered real help and guidance in a genuine work environment. Our
testers found it to be highly engaging and played on it for a very long time, determined to beat
the game!
Again, most of the points and issues raised during usability testing have been resolved and the
current piloting version is already being extremely well received by refugees, trainers and
employers.