L.E.M.E.S.
Project Presentation (P)
Context:
Evtek, User Interfaces course by Marco Sentinelli, 11.8.2006
Team Name:
L.E.M.E.S. (1)
Team members:
Natalya Kulishenko (0401046)
Lilia Galyautdinova (0500603)
Julia Moltchanova (0500617)
Joost Oostdijk (0300776)
Hristiyan Dimov (0300781)
Esa Hakkinen (0102257)
Enrico Taddeo (0200681)
(1)
from Learnability-Efficiency- Memorability-Error-Satisfaction
Table of contents
1 - Introduction
1.1 About the project
1.2 Products description
2 - Planning phase
2.1 Target users and tested users
2.2 Environment
3 - Testing phase
4 - Analysis phase
4.1. Quantitative analysis
4.2 Qualitative analysis
4.3 Top usability problems
5 - Improvements
6 - Conclusions
1 - Introduction
1.1 About the project
The aim of this project is to learn to be proficient in analyzing and comparing to each other three or more
products (chosen from a proper set of similar products) from an usability perspective in order to be able to
effectively evaluate their user interfaces thus gaining the ability to achieve a more reliable description/review of
the product and the knowledge necessary to make appropriate proposals for improvements.
1.2 Products description
The test product we have chosen is non-professional digital photo camera.
All the cameras are of the same price and quality levels and also have many features for testing.
Cameras have at least interfaces in English language.
1.3 Insight into products and their market
Hugest camera market is in “general-purpose” camera market area. World wide sales are concentrated in
cameras that are easy to use and they should be cheap compared to the features and quality of pictures. Users
expect that camera will be so simple to use, that they can just dig it from pocket and shoot the picture without
thinking lightning conditions or flash duration. Digital cameras are technology products and are often more
hard to use than previously existing analog cameras with just trigger and handle to rewind the film; they have
anyway many more features to make pictures more vivid and give owners more possibilities to succeed in
performing advanced with less troubles.
In our project the cameras were selected from our own cameras, as we found them to represent one unified
product range, actually the generic camera market. Although three cameras were in same 300-400 EUR price
and feature range, their outlook was still quite different from each other. Minolta looks like professional camera
with classical systems camera, HP looks really slick and easy to use without too many fancy buttons, and
Canon is something in between combining user interface ideas from both.
2 - Planning phase
2.1 Target users and tested users
We tried to found test users who have not used digital cameras before. As well some of them should have better
knowledge about lightning conditions and some basic experience on taking a basic picture, how to operate and
keep a camera in their hands. And again others shall have never kept even an analog camera in their hands. As
these cameras represent average mass market, the end users should represent cautious, low risk taker kind of
personalities, more than technological enthusiastic or professional photographer.
The actual users we were able to bring together in the real end were:
 Four undergraduate students
 A PhD student
 A researcher
 A kindergarten teacher
Test was conducted in the university (where 3 of the users did perform the test) and also at user’s premises
(four of the users did it in their home environment). Among the users, 5 used digital camera before and 2 did
not. For all users English was not native language. The users were chosen among people we did know.
2.2 The environment
The optimal environment is the one which is not having any influence on the user under test. For instance, the
observer should not be visible and the surroundings should be familiar with those the user would find himself
in, when performing the simple actions requested.
In theory the user should naturally find the camera at the time he needs it (willing) and do his best
(comfortable) to achieve the goal of taking a picture at the latest extent (satisfaction) without external help and
in a reasonable time.
The theoretical environment was impossible to recreate due to limitations in resources (time to travel
availability as a primary reason). Moreover the test was arranged in such a way that the observer must be
present and interactive, being able to: -request operations, -observe and collect behavioral and procedural data, wait for comments (talk aloud), -be ready when demand for help is needed (in order to try to limit the critical
errors leading to an avoidable no success end status), -limit the time given to perform tasks.
To limit the influence of the observer (found to be main source of pressure) the following measures were taken
to proceed with the tests:
 before the test phase the users were briefly introduced into the test’s idea, goals, the tasks they
were about to be invited to perform and they were asked a first set of formal questions like name
and age with the intention of creating as well a friendly atmosphere.
 they were shown the ready prepared form (please, see attached pdf file) in order to make them
confident that the same procedures are taken for all the tested users according to standardized
procedures that the observer is going to follow no matter of inconveniencies as well as previous
failures.
 The observer is supposed to be silent and participate only if help is explicitly requested or when
the time assigned to each task (five minutes) is over.
Also no one else except testers and tested persons were accepted to participate or assist during the examination
to avoid user being distracted by external comments or any event in general.
At this point it is convenient to anticipate an analyzed aspect: when the users were invited to perform the test 35% of
them were caught in a hurry and after a long day which corresponded in tiredness and low willingness to do it.
3 - Testing phase
Despite our initial intentions in accordance to our plan which was written down in very general format, we went
through more detailed and better described steps when the real test was finally performed. Let’s see the steps:
 The users are introduced with an overview of the whole program and, following, a couple of
questions are submitted regarding personal profile.
 Cameras are introduced as similar in most of their aspects.
 User is given the imaginary opportunity of choosing one camera for himself as a gift after the
test is done but the choice is based upon the feeling of easiness that the camera inspires.
 Then one task at a time is asked (in a form of a real time situation) to be executed within 5
minutes without intervention of observer unless explicitly required. The order of the task is
chosen to be all five in a row for HP then the next five concerning Canon and finally the five
for Minolta.
 Observer is sitting in front of user taking care to move as less as possible and avoiding
comments between one test and the other, provided it was made clear that on the contrary the
user should try to comment loudly like if he was alone dealing with the challenges of the
camera.
 After tasks are performed, the observer congratulates with the user, thanks him and ask for a
last effort, filling the questionnaire which is planned and written to be as more
comprehensible as possible (please, see again attached pdf file).
Task scenario
Next are the real time situations proposed to the users as a starting point for the single tasks:
1.
“Your batteries are low, please change them / Please, remove the memory card since transfer of
pictures to pc is required.”
2. “Before using digital camera, you need to know its modes. Please, define the camera modes as you can
understand by reading the icons printed in the top dial”.
3. “If the camera is originally set in English and you want to have it in your native language, you need to
know how to do it. Please, set Finnish language if present.”
4. “In case you want to take a picture of yourself, you need to use Self-Timer mode. Please, set the camera
to Self-timer mode.”
5. “Using your camera you will be taking pictures in different environments. Please, make a picture in the
dark using flash and red-eye reduction.”
4 - Analysis phase
4.1 Quantitative analysis
According to established aim of our test to evaluate usability we measured success rate of accomplished tasks
and average time for completing each task.
From the collected data we counted the amount of successfully completed tasks. HP digital camera has got the
highest success rate (88 %). On the second place after HP came Canon (68 %) and Minolta (60%) has got the
least success rate. The result of this survey reviled that HP digital camera is the easiest in use.
By counting average time for completing each task we discovered that less time was needed for completing task
using HP digital camera. On the second place was Canon digital camera and Minolta was the most time
consuming digital camera for performing our tasks.
Analyzing all tasks by time performed by each digital camera we could mark out:
 HP digital camera: as the most usable in the task of changing batteries, showing the less time for
completing this task.
 The Cannon camera: as the most usable in the task of changing language, showing the less time
for completing this task.
 The Minolta camera: here it did won the special award for the worst usability performance.
4.2 Qualitative analysis
During performed tests observer tried to make as many notes to describe the users’ behavior as possible.
Observer tried to figure out from test user body language and emotions. We choose 4 criteria to distinguish the
emotions during tests: willingness to perform test, level of confusing during performing tasks, level of felt
comfort during performing tasks, fear to make a mistake.
Willingness to perform tasks:
How many tasks were performed with really noticeable users’ willingness?
Result: Minolta has the lowest percentage because of it’s professional-like appearance. Many users were afraid
of operating such a professional (complex) thing.
Level of confusing during performing tasks:
How many tasks cause users’ confusion?
Result: HP got the lowest percentage, because it doesn’t have so many menu items; it just does not give a
possibility to go wrong.
Felt comfortable during the tests:
During how many tasks users felt quite comfortable?
Result: Users felt more comfortable dealing with HP. The simplest camera doesn’t force users to think a lot and
allows them to feel themselves relaxed.
Afraid to make a mistake:
How many tasks cause a fear of making mistakes?
Result: Canon caused the most feeling of fear. Canon has weak mechanical door design which raised above
others.
Based on results of observation during performing tests and users answers from questionnaire it was possible to
figure out the main list of usability problems.
4.3 Top usability problems (number of opinions):
HP
 flash light button is hard to locate (2)
 button to enter into menu is not obvious (1)
 timer icon is no very obvious (1)
Canon
 It’s hard to realize for beginners the meanings of the icon used on programming button (7)
 Mechanical design of cover’s is hard to use (to learn). (5)
 Menu is not easy to use: difficulties to go from horizontal tab to another tab (2)
 On the menu users can’t connect icons with their explanation (red-eye icon) (1)
Minolta
 Nobody did manage to find out how to set the self-timer shot (7)
 Nobody could understand the huge amount of symbols on the programming button (7)
 Menu is complex: too much stuff on the screen (6).
 Users were afraid to use too much efforts to close the battery door (4)
 Setting the flash is hard to locate: users didn’t expect to find it on top of the camera (4)
5 - Improvements
“The camera is an image-expression tool, an extension of your hands and eyes, which is why the ideal
camera is designed to meet your needs and preferences”. (Canon.com)
During evaluation of testing results we were revealed an amount of problems with our digital cameras. Those
were mechanical as well as technical. We marked out several problems which appeared to be more common
and made suggestions on them.
HP photosmart 812
Flash setting
Several testers have faced difficulties, finding a flash settings button on the camera. Our analysis has shown
that the current location for this button is not intuitively simple for the users and blends among other settings on
the back end of the camera. Our suggestion is to move this button on top of the camera, closer to shooting
button, where it will stand out and catch user’s attention.
Buttons placement
Several users have complained about placement of buttons on the camera. For them, one of the advantages of
small camera is possibility of browsing buttons with one hand. However, the number of such users is much
smaller then those, who prefer to operate the camera with both hands. Perhaps, a separate product may target
single-handed operation mode.
Canon PowerShot A75
Icons on programming button (Exposure Mode Dial)
None of our users managed to name all the icons on the programming button (Exposure Mode Dial as seen in
the picture above)
Testing photo cameras is quite a challenging task, since it doesn’t involve only camera’s user interface design,
but it as well requires some general knowledge of photography. For instance, a person who ever owned a midlevel digital camera would have much better idea of the markings and possible menu-settings, when compared
to a person, who has never touched such device. As a result, during our tests, few users have complained that
there are too many shooting modes available on the main dial. If most of the modes are marked with quite
recognizable pictures (for night shots, portraits and motion shots), some others are marked abbreviations, which
are impossible to guess. However, such dial is the fastest way to change modes of operation.
Mechanical design of covers
Some users have found design of the camera’s body itself difficult to use. Especially, cover for memory card
and butteries. Though, this mechanical part is a subject to different requirements, where mechanical durability
and reliability come before user friendliness.
Minolta DImage Z2
Flash
Flash. The brilliance of flash design of this camera amazes most new and quite a few experienced users. It has
to be open by HAND. Even more confusing in this situation is the button, marked with “flash” sign. User
presses it – nothing happens. Nothing will happen, until user opens a bridge, containing a flash. Then button
starts to operate. Obvious improvement suggestion: flash has to open automatically, according to button
settings. Adjusting the flash bridge to the closed position may still be manual.
Some testers have also commented on the location of the flash button. It is in unexpected place on top of the
camera. However, we still would opt for such solution, since otherwise, the button my blend between other
controls as it has happened with HP camera.
Battery’s cover
This camera have a very loose battery cover mechanism, which requires quite a lot of force to move it to proper
closed position. In many cases, users are becoming concerned if they will break it mechanically. Although, this
mechanical part is a subject of mechanical durability and reliability.
Self-timer mode
One of the often used features of a digital camera is self timer. So, we have asked out testers to try to use this
feature. Konica-Minolta has shown the worst performance on this issue. Even its menu’s depth is 2, the name of
the sub-menu, which contains self-timer setting, is confusing (sounds to complex for a new users), hence they
we wondering to look for the feature is far deeper corners of the camera’s setup. Even when user is able to find
the correct menu setting, he is being puzzled by automation, which is build-in the camera. Camera’s settings
will indicate that a self-timer mode is now set, but no timing information will be displayed. Camera starts
automatic count-down of 10 seconds as soon as you press “shoot” button. As an improvement to the current
design, we propose to make menu more explanatory, by showing the actual count-down time, in the submenu
where selection is made.
Complex menu
To a user who has little experience this particular camera has one of the best menu designs among our test
subjects. Its menu is divided into two parts, one of which covers imaging options and the other is responsible
for camera’s technical setup. First part of the menu is designed easy with 2-level settings. Setup part is much
more complex, but it is not supposed to be used often and only after consulting with user manual. However, our
tests has revealed that first menu is lacking clarity so users tend to go looking into the setup portion, where even
an experience user might get lost in all the information displayed. One also may argue that menu design
approach, where rather then scrolling the screen, several tabs are used to display information is not very
intuitive. Though, there is no single right solution for this problem, since the amount of information, which has
to be access via this menu is vast. Several improvements may apply to naming menu items more clearly (like in
case with self-timer) and to add more feedback to the user, when certain settings are activated, by the use of
pop-up windows.
Unmarked buttons
Finally, there are few completely unmarked buttons of the Four-Way Arrow Pad and OK Button. Even though,
their mutual arrangement and position intuitively associate these buttons with menu navigation, some marking
would be an improvement. Especially, for the middle button, acting as a selection trigger.
6 - Conclusions
Results
Most visible and important result, apart from the statistical data previously made available, was finding how our
research data could clearly reveal two main facets: usability as measured by the overlapping of accessibility and
availability (that is possibilities and demand) and the misleads that aesthetics brings together when trying to
estimate ergonomics and efficiency at first glance (Minolta). That said, when we come to practical conclusions
focusing on the products themselves we find that they should be made more simple, less stuffed with
unnecessary features just for the sake of marketing, more quick and respondent about those main purposes a
camera is created for, as it is clear the user expects only what he needs and his needs are usually basic and
obvious.
The testing
Here is the slogan for our project: ”You get what you measure”. Among all difficulties in creating an iterative
piloting within test planning and test phase, the final objectives were achieved: we got enough and coherent
data to measure quantitative and qualitative aspects in the terms of usability and search for improvements to fill
the goals we set on our own. Room for improvement is found in the possibility to redesign the testing
procedures, feasible in this particular case only with having the opportunity to experience a new project of the
same kind.
The teamwork
As said, we had to set our own goals as it was clear since the beginning of this adventure that team members
were hard to melt themselves in a group with a supposedly clear target which was not clear at all. The main
problem we faced was a communication problem inside a team made of excellent members who weren’t able to
divide properly their duties, but had to conquer their position in a clumsy-established framework. Being forced
to establish leadership helped us in returning on track and our final product was all in all in line with the course
requirements.