6th INTERNATIONAL CONFERENCE ON
QUALITY SOFTWARE (QSIC 2OO6)
Beijing, China - October 27-28, 2006
Early Usability Evaluation in Model-Driven
Architecture Environments
Silvia Abrahão and Emilio Insfrán
Departmento de Sistemas Informáticos y Computación (DSIC)
Universidad Politécnica de Valencia
{sabrahao, einsfran}@dsic.upv.es
1
Contents


Introduction
A Framework for Usability Evaluation
Usability at the Platform Independent Model (PIM)
 Usability at the Platform Specific Model (PSM)





Usability Model
Instantiating the Usability Framework in a MDA tool
Results
Conclusions and Further Work
QSIC 2006, Beijing, China - October 27-28
2
Introduction




Software development companies recognize the usability as a key
factor for the success of their products.
However, software engineering practices don’t consider usability
aspects properly.
Several studies show that 80% of total maintenance costs are
related to problems of users with the system. Among these costs,
64% are related to usability problems! (Landauer95)
Many approaches to evaluate the software usability have been
proposed in the last years. Most of them focus on defining a set of
attributes that explains usability and on developing guidelines and
heuristics for testing it.

They mainly focus on artifacts used in the late stages of software
development without connecting usability evaluation results with
changes in software design.
QSIC 2006, Beijing, China - October 27-28
3
Introduction

Model-Driven Software Development (MDSD) appears with its main
characteristics:


Separation of business functionality from technology implementation
The use of models along the complete software development process
by means of transformations (until the final source code)
This is a new challenge for the usability evaluation!

There are many tools and approaches for MDA (including User
Interface specification) but none of these incorporate the usability
explicitly

There are no approaches which systematically evaluate the usability
of the applications obtained as a result of a process of successive
transformations.
QSIC 2006, Beijing, China - October 27-28
4
Contents


Introduction
A Framework for Usability Evaluation
Usability at the Platform Independent Model
 Usability at the Platform Specific Model





Usability Model
Instantiating the Usability Framework in a MDA tool
Results
Conclusions and Further Work
QSIC 2006, Beijing, China - October 27-28
5
A Framework for Usability Evaluation
Goal

A generic framework that incorporates usability in MDA
processes, where:

The usability of a software application is evaluated and improved at the
platform independent model (PIM) using a Usability Model.

As outcome, a list of usability problems is produced.

Instead of relate a given usability problem with changes in the interface
tier (code model level), we relate it to the elements of the PIM that are
affected by it.

The evaluation also gives a feedback to the transformation rules itself
(PSM).
QSIC 2006, Beijing, China - October 27-28
6
A Framework for Usability Evaluation
REQUIREMENTS
ELICITATION



An iterative process that
combines model development
and usability evaluation. It is
applied until the PIM has the
required level of usability.
The evaluation is made
without users.
This allows very early
usability evaluation by
verifying the PIM properties
that are related to usability.
COMPUTATION
INDEPENDENT
MODEL
REQUIREMENTS
MODEL
SYSTEM
ANALYSIS
CONCEPTUAL MODELS
1A
Usability Model
OBJECT MODEL
PLATFORM
INDEPENDENT
MODEL
TASKS MODEL
DIALOG MODEL
PRESENTATION MODEL
PLATFORM
INDEPENDENT
USABILITY
REPORT
EVALUATION
WITHOUT USERS
...
MODEL
PLATFORM
SPECIFIC
MODEL
COMPILATION
APPLICATION
MODEL
1B
SOURCE CODE
CODE
MODEL
INTERFACE TIER
APPLICATION TIER
PERSISTENCE TIER
MDA proposal
MDA-based
development process
QSIC 2006, Beijing, China - October 27-28
Early Usability evaluation
process
7
Contents


Introduction
A Framework for Usability Evaluation
Usability at the Platform Independent Model
 Usability at the Platform Specific Model





Usability Model
Instantiating the Usability Framework in a MDA tool
Results
Conclusions and Further Work
QSIC 2006, Beijing, China - October 27-28
8
Usability Model
Goal


The development of the model followed several activities.
According to the Goal-Question-Metric (GQM), the goal of our
usability model is:

Analyze
For the purpose of
 with respect to its
 from the viewpoint of
 In the context of

PIM (task, dialog, presentation
models, etc.) and/or the CM (final UI)
Evaluate
Usability
a set of evaluators
Software applications obtained with a MDA
development process
QSIC 2006, Beijing, China - October 27-28
9
Usability Model
Specify Sub-Characteristics and Attributes


This activity is concerned with the decomposition of Usability into a
set of sub-characteristics and measurable attributes.
The ISO/IEC 9126-1 quality model is used as the basis for building
the proposed usability model:
ISO 9126
Functionality
Reliability
Usability
Efficiency
Maintainability
Portability
Learnability
Understandability
Operability
Attractiveness
Compliance
Concepts with high level of abstraction
QSIC 2006, Beijing, China - October 27-28
10
Usability Model
Specify Sub-Characteristics and Attributes

The decomposition is done to support usability inspection methods
and to identify and fix usability problems.

1-Learnability refers to the attributes of a software product that
facilitate learning. In our model it is defined in terms of:

Help facilities (pre-defined keywords, wizards, on-line help, and
documentation)

Predictability which refers to the ease with which a user can determine
the result of his/her future actions,

Informative feedback in response to user actions

Memorability as a measure of how quickly and accurately users can
remember how to use an application that they have used before.
QSIC 2006, Beijing, China - October 27-28
11
Usability Model
Specify Sub-Characteristics and Attributes

2-Understandability refers to the attributes that facilitate
understanding :

We differentiate the own optical legibility of texts and images from
readability that refers to the information grouping cohesiveness and
density.

Familiarity is the degree to which a user recognizes UI components and
views their interaction as natural; UI can be familiar by imitating the
visual appearance of real-world objects, by relying on standardized
commands, or by following other common metaphors.

Brevity is related to the reduction of user cognitive effort (i.e., limiting
the reading and input workload and the number of action steps).

User guidance is related to the message quality and navigability to
guide the user by providing mechanisms such as path and current
position.
QSIC 2006, Beijing, China - October 27-28
12
Usability Model
Specify Sub-Characteristics and Attributes

3-Operability refers to the attributes that facilitate user control
and operation:

Capacity of the system to provide instalability facilities (e.g., user’s
help, documentation and installation procedures)

Data validation (data validity), the user’s degree of control to services
execution (controlability),

Capability of adaptation of the UI or services. It refers to the system
capacity to behave contextually and according to the users’ needs and
preferences. We distinguish between


Adaptability, which is the system capacity to be adapted by the user, and

Adaptivity, which is the system capacity to adapt to the users’ needs.
Operational consistency in the execution of services and controls
(consistency), the prevention and error management, and the capacity
to monitor the state of the system (state system monitoring).
QSIC 2006, Beijing, China - October 27-28
13
Usability Model
Specify Sub-Characteristics and Attributes

The other two sub-characteristics are related to the perception of
the end-user (4-Attractiveness) or evaluator (5-Compliance) using
the system. It can be mainly measured using subjective measures
(e.g., surveys).

Attractiveness: some aspects of attractiveness related to aesthetic
design can also be quantified by measuring the UI uniformity in terms
of font color, font style, font size and elements position.

Example: Some studies suggest that designs should not use more than 23 fully saturated intense colors.
QSIC 2006, Beijing, China - October 27-28
14
Usability Model
Specify Sub-Characteristics and Attributes

Compliance: can be measured by assessing the agreement of the
proposed usability model with respect to the following standards:
ISO/IEC 9126-1 (from SE field)
 ISO 9241-11
(from HCI field)
 Microsoft and Java style guides
 …


The first two assessments can be performed at PIM or CM levels,
whereas the last one can only be done at the CM, as it analyses the
fulfillment of the UI widgets to the prescriptions of the style guides.
QSIC 2006, Beijing, China - October 27-28
15
Usability Model

Learnability

Help Facilities


Documentation
Completeness
Multi-user Documentation
















Font Size
Contrasting Text
Disposition
Labeling Significance
Internationalization
Metaphor
Brevity

Values Inizialization


Time to Remember
Accuracy
Legibility
Information Density
Workload Reduction


Understandability

Location Grouping
Format Grouping
Familiarity

Icon Significance
Icon/Link Title Significance
Action Determination
Information Grouping Cohesiveness

Informative Feedback
Memorabitity


Capability Profiled
Role Profiled
Readability

Predictability





Initial Values Completion
Initial Values Modifiability
Actions Minimization
Self-descriptiveness
User Guidance

Message Quality

Navigability
QSIC 2006, Beijing, China - October 27-28
16
Usability Model

Operability

Installability












Adaptability
Adaptivity














State System Monitoring
Background Color Uniformity
Font Color Uniformity
Font Style Uniformity
Font Size Uniformity
UI Position Uniformity
Subjective Appealing
Compliance

Steady Behavior of Controls
Permanence of Controls
Stability of Controls
Order Consistency
Label Consistency
Error Prevention
Error Recovery
Attractiveness
Consistency

Error Management

Edition Deferral
Cancel Support
Explicit Execution
Interruption Support
Undo Support
Redo Support
Capability of Adaptation


Ease of Installation
Multiplicity of Installation
Updateability
Update Transparency
Data Validity
Controlability





Degree of Fulfillment with
the ISO 9241-10 / ISO 924111
Degree of Fulfillment with
the Microsoft style guide
Degree of Fulfillment with
the Java style guide
QSIC 2006, Beijing, China - October 27-28
17
Contents


Introduction
A Framework for Usability Evaluation
Usability at the Platform Independent Model
 Usability at the Platform Specific Model





Usability Model
Operationalizing the Usability Model into a MDA tool
Results
Conclusions and Further Work
QSIC 2006, Beijing, China - October 27-28
18
Operationalizing the Usability Model into a
MDA tool

The Usability framwork was
instantiate in a specific MDA tool:
OlivaNova Model Execution (ONME)
Service IU
Introduction
Defined Selection
Argument
Grouping
Dependency

This tool is based on:


Conceptual Model (PIM): Object
Model, Dynamic Model, Functional
Model and Presentation Model
Model Compilers that establish
the specific to a given platform
(PSM) and generates the code (CM)
applying transformation rules
Supplementary
Information
Hierarchical Action
Tree
Population IU
Filter
Order Criterium
Instance IU
Display Set
Actions
Navigation
Master/Detail IU
Key
A
B
A uses B
QSIC 2006, Beijing, China - October 27-28
19
Operationalizing the Usability Model into a
MDA tool
Usability

Association of one or
more elements of the
PIM / CM models to
each identified
attribute.

The attributes in our
Usability Model are
inherent to UI
modeling and make a
product usable.

One or more metrics
are associated to
each model attribute.
Platform Independent Model (PIM)
ISO/IEC 9126
Usability
definitions
Ergonomic
criteria
Operability
Learnability
Consistency
Compliance
explicit control
Error
Management
Prompting
Information
Significance
Explicit Control
Attractiveness
Understandaility
Legibility
Workload
ONME Presentation patterns
Abstract
interface
patterns
HAT
AJA
Master-Detail
IU
UI de maestro/detalle
UI de instancia
Instance
IU
UI de poblaci
ón
Poblation
IU
UI de servicio
Service
IU
A/Z
F
Filter
N
Display set
A
Navegation
Action
Order criteria
AIO (Abstract Interface Objects)
Platform Specific Model (PSM)
Concrete
interface
components
Visual Basic
Java/
Swing
ColdFusion
PocketPC
ASP
Computing Platforms
Choose of
platform
Menu
Explore
Bar
Grid
Control
Filter
Generic
component
INSTANCE
Radio
Button
Interface
standards
- IBM CUA
- Microsoft
...
JSP
CIO (Concrete Interface Object) VB
QSIC 2006, Beijing, China - October 27-28
20
Contents


Introduction
A Framework for Usability Evaluation
Usability at the Platform Independent Model
 Usability at the Platform Specific Model





Usability Model
Instantiating the Usability Framework in a MDA tool
Results
Conclusions and Further Work
QSIC 2006, Beijing, China - October 27-28
21
Results

Examples of sub-characteristics, attributes, PIM elements affected
and potential metrics.
QSIC 2006, Beijing, China - October 27-28
22
Results Obtained and their Effects on the PIM
and PSM

The Usability Model was applied across several usability
inspections carried out in ONME interface models (PIM) and
generated applications (CM).

This allowed us to validate the model experimentally.

Some identified Usability Problems (UPs):






Lack of descriptive labels
Difficult navigation
Inflexible/invalid search capabilities
Inconsistent menu item and window labels
No feedback for user selection
Meaningless error messages
QSIC 2006, Beijing, China - October 27-28
23
Usability Problems that affect the PIM


Once identified the usability problems, its source must be
determined (PIM or PSM) to define a Recommendations Plan.

e.g. the problem Meaningless error messages affects the quality
of error messages

Although the message informs how to solve the problem, they
must be meaningful for users!
In this case, the source of the problem is in the PIM
(Presentation Model) and relates to the property “Validation
message” of the Introduction Pattern of a Service Interaction
Unit (Service IU).
QSIC 2006, Beijing, China - October 27-28
24
Recommending a change in the PIM
Identifier:
17
Usability Problem:
Lack of descriptive labels
Context:
Instance Interaction Unit (IIU) and Supplementary Information pattern
Related ergonomic
criteria:
Guidance
Usability attribute:
Labeling significance
Example:
Note that the display set of the IIU (indicated by the row) has no label.
Recommendation:
Identify the patterns in the Presentation Model without alias and define
them.
Apply the proposed metric (proportion of elements with meaningful
names).
QSIC 2006, Beijing, China - October 27-28
25
Usability Problems that affect the PSM

Information grouping cohesiveness and information density
attributes were poor evaluated because of the lack of grouping
mechanisms. This problem and recommendation is as follows:
Identifier
10
Usability Problem
The users’ understanding of a display screen depends on the grouping
and density of the objects (images, texts, etc.) that are presented.
Currently, in ONME, all the attributes are shown in an abstract
interaction unit without any grouping mechanism.
Context
Instance Interaction Units, Master-Detail Interaction Units
Related ergonomic
criteria:
User explicit control, Predictability
Usability attribute:
Information group cohesiveness and Information density
Recommendation:
Change the transformation rules (PSM) to incorporate different
representations to the grouping mechanisms such as boxes, tabs, eye
fishing, etc.
QSIC 2006, Beijing, China - October 27-28
26
MDA Benefits

In a MDA environment, some usability attributes are garantee by
the transformation process
 User guidance attributes




Legibility attributes:




Visual feedback in menus and dialogs boxes where choices can be
selected or when cursor is pointing (tooltips)
Visual feedback when elements (objects) are selected
…
Prompts and error messages always appear in the same place(s)
Error messages displayed in pop-up windows with the same structure:
what-where-howToSolveIt
…
These give us empirical evidence of the intrinsic quality of UIs
generated by a MDA tool.
QSIC 2006, Beijing, China - October 27-28
27
Contents


Introduction
A Framework for Usability Evaluation
Usability at the Platform Independent Model
 Usability at the Platform Specific Model





Usability Model
Instantiating the Usability Framework in a MDA tool
Results
Conclusions and Further Work
QSIC 2006, Beijing, China - October 27-28
28
Conclusions

We have presented a framework for early usability evaluation
that fits in MDA development processes.

Unlike existing proposals, the usability is evaluated and
improved at the PIM level instead of in its final user interface
(CM).

This framework has been operationalized by means of a
Usability Model which represents the relationships among the
ISO/IEC 9126 and the usability patterns and criteria from the
HCI field.

This model was applied in the inspection of several PIMs
(presentation models) specified using a MDA tool.
QSIC 2006, Beijing, China - October 27-28
29
Conclusions

The framework was found to be very useful not only to evaluate
PIMs but also to discover deficiencies and/or limitations of the
PIM expressiveness and the transformation rules (giving feedback
for their improvement!).

The contribution of the work is to provide a set of attributes
(independent of any method or technology) that can be verified
in the PIM/PSM/CM defining the corresponding relationships and
metrics.
QSIC 2006, Beijing, China - October 27-28
30
Further Work

The application of the usability framework to other MDA tools
specifically oriented to Web development (i.e., VisualWade).


We are concerned, in particular, to explore and evaluate to which
extend our usability model can improve the development of Web
applications.
The operationalization of the Usability Model using a set of
usability metrics from the Web Quality Model (Calero et al.,
2005). The first project will be the evaluation of the usability of
a Spanish museum (Thyssen-Bornemisza Museum) from its code
model.

This will allow to measure the level of usability of a model or a
implemented UI.
QSIC 2006, Beijing, China - October 27-28
31
Thanks for your attention!
QSIC 2006, Beijing, China - October 27-28
32