Universiteit van Amsterdam
Faculty of Science
Enhancing productivity of Océ Field
Service Technicians through
visualization of relevant data
Master’s Thesis
Information Science, Business Information Systems
Konstantins Babahodzajevs
Student ID: 10426264
8/13/2013
Supervisor: Tom van Engers
First examiner: Tom van Engers
Second examiner:
Contents
1
Introduction ..................................................................................................................................... 5
1.1
Context of use ......................................................................................................................... 6
1.1.1
Who is Field Service Technician (FST) and what information does he need to be
successful? ....................................................................................................................................... 7
1.1.2
1.2
2
Data type ......................................................................................................................... 7
What is data visualization? ...................................................................................................... 9
Conceptual Framework ................................................................................................................. 10
2.1.1
Exploration of visualization strategies........................................................................... 10
2.1.2
Determining strategies for the given case .................................................................... 11
2.1.3
Test of chosen strategy ................................................................................................. 11
2.2
Decision-making .................................................................................................................... 11
2.2.1
CARE concept................................................................................................................. 13
2.2.2
CARE concept and decision-making .............................................................................. 13
3
Literature review ........................................................................................................................... 14
4
Interviews ...................................................................................................................................... 16
4.1
Approach ............................................................................................................................... 16
4.2
Results ................................................................................................................................... 17
5
Design of new visualization strategy ............................................................................................. 17
6
Test and evaluation ....................................................................................................................... 19
6.1
SWOT of new strategy ........................................................................................................... 23
7
Results ........................................................................................................................................... 24
8
Conclusions.................................................................................................................................... 26
9
Future Work .................................................................................................................................. 28
Appendix 1............................................................................................................................................. 31
Current STAR version............................................................................................................................. 31
Service Tooling for Analysis and Reporting (STAR) ........................................................................... 31
SMART ........................................................................................................................................... 31
Analysis .......................................................................................................................................... 32
Errors and warnings....................................................................................................................... 32
Algorithms for pattern recognition ............................................................................................... 32
Appendix 2............................................................................................................................................. 33
Interview 1............................................................................................................................................. 33
Introduction....................................................................................................................................... 33
2
Description of approach ................................................................................................................ 34
Summary ........................................................................................................................................... 35
Proposals ........................................................................................................................................... 36
Summaries of interviews ................................................................................................................... 39
FST 1 .............................................................................................................................................. 39
FST 2 .............................................................................................................................................. 39
FST 3 .............................................................................................................................................. 39
FST 4 .............................................................................................................................................. 40
FST 5 .............................................................................................................................................. 40
FSTs 6/7 ......................................................................................................................................... 40
FST 8 .............................................................................................................................................. 41
Interviews 2 ........................................................................................................................................... 41
Appendix 3. Description of the prototype (to remain within Océ) ........... Error! Bookmark not defined.
Appendix 4. Description of task for a test (to remain within Océ) ........... Error! Bookmark not defined.
Appendix 5. Definitions of criteria used in evaluation model ............................................................... 42
10
Bibliography............................................................................................................................... 43
Table of figures
Picture 1”Visualization process” (Chen M., 2009) .................................................................................. 9
Picture 2 “Data-Reader-Designer-Trinity” (Iliinsky N., 2011) ................................................................ 10
Picture 3 “Worklist” ............................................................................................................................... 31
3
Abstract:
In a modern business environment, technology is a driving force. A
large amount of processes have become fully or partially automated.
Equipment and software used to automate processes require regular
maintenance. In order to perform effective maintenance, information about
the performance of the product is necessary. Data that might be used for
maintenance purposes can be extracted from the product, but that data is
stored in a format that can hardly be processed by a human brain. Therefore
a translation is needed that would change data into information.
In last 20 years emphasize has been put in researching data
visualization techniques. Different approaches are used depending on type,
size and purpose of data as well as type of user and his/-her requirements
and demands for the information. This master thesis describes a study
conducted as part of an internship within Océ which had the purpose to
enhance productivity of Océ Field Service FSTs by designing a visualization
strategy that would provide relevant information from the data
accumulated in Océ products.
4
1
Introduction
Data is a result of every technological process. It is a way of understanding what is happening
within those processes, determine quality of performance of equipment, such as finding it’s wearing
out parts, finding possible opportunities for improvement etc. But data is a computational output,
which makes it extremely difficult for human brain to process.
Visualization of data is a process of making data readable for human, turning data into
information. There are multiple strategies and techniques that might be used to turn data into
information depending on the type and size of data as well as type of reader of that data (Iliinsky N.,
2011) .
Océ Technologies B.V. a Canon Group Company, is a Dutch company, headquartered in Venlo,
Netherlands that develops, manufactures, sells, and supports printing and copying hardware and
related software. The offering includes office printing and copying systems; production printers and
wide format printing systems for both technical documentation and color display graphics
(http://en.wikipedia.org/wiki/Océ , 29.03.2013 22:25). Usage of printing hardware and software
results in large amounts of accumulated data on performance of hardware, which can be used to
produce information on what maintenance activities are required to be taken now, and which issues
and risks are potentially possible. This data has to be retrieved and visualized in a way so that Field
Service Technicians (FST), Océ employees responsible for providing service in terms of system
installation, corrective maintenance, preventive maintenance, modification integration and operator
training for the standard products, might take necessary actions as fast as possible being efficient
and effective.
Current software used by Océ FSTs for exploration and explanation of data has been developed
over 15 years ago. Since 1990s data visualization strategies have evolved significantly. Knowledge
base on how to build more effective and efficient data visualizations is now deeper and tools
designed for transformation of data into information are able to perform much quicker, what allows
users to perform interactive searches faster and process bigger amounts of data (Chen M., 2009).
Over last two decades there was a lot of emphasis put on these issues, so there certainly is a room
for improvement of current approach used by Océ to visualize data.
But there is another dimension that must be considered when investigating given case. FSTs are
making certain decisions while performing maintenance activities. These decisions are based on the
information provided by the visualization of data extracted from the product and provided by the
customer and affect the overall efficiency and effectiveness of the service visit. Current tooling used
for data visualization, STAR, serves as a display for data and as an advisory tool, that provides a list of
possible issues that FST should consider, which implies that FSTs have to make sense out of it mostly
by themselves, which is resource consuming and unsynchronized, meaning that every FST potentially
can make different decisions depending on his working method and knowledge. In current research I
will attempt on investigating upon relation between visualization and decision-making, as a basis
taking the bounded rationality theory: How can data visualization be used in order to make decisionmaking of FSTs more efficient and effective? By efficiency and effectiveness in the context of current
5
research I refer to how fast, complete and satisfactory for client as well as for the company are the
maintenance activities of FSTs.
Research performed within this assignment will also focus on finding most suitable strategy that
would help Océ to visualize data extracted from equipment and adopting it. The proposals developed
throughout this assignment are aimed on enhancing maintenance activities performed by FST’s by
providing them with relevant information presented in easy readable and understandable way.
A lot of research has been done on data visualization in the past years, especially for the last few
years with emerging concerns over big data. Current research addresses visualization of data aimed
on assisting FST’s which has to be practical and to-the-point rather than esthetically advanced, which
is the case for the most of visualizations (Iliinsky N., 2011)
With my research I aimed on expanding knowledge on how to develop data visualization for
FST’s. I have conducted multiple rounds of interviews as well as analyzed current tooling using
criteria proposed by Smith and Mosier (Smith L.S., 1986) with a purpose to define requirements for
new visualization; developed a PowerPoint prototype and applied cognitive walkthrough method in
combination with think aloud protocol to test it with expanded System Usability Scale, further SUS,
evaluation model to determine whether FSTs find the proposal as an improvement. This document
might be used as a guideline for developing visualization of data for technical maintenance purposes
for Océ as well as other organizations with similar interests as well as provide a case study in support
of theories concerning decision-making, such as bounded rationality and its developments.
With the knowledge available now and evolved technologies, such as programming languages,
faster computers, introduction of touch screens, the service tooling used now by FST can certainly be
improved through multiple ways that are described further in current paper. Many studies have been
conducted and developments have taken place over last 15 years and there are many techniques
that could be applied for improving current tooling and certain techniques have been chosen, but not
a lot has been studied on the relation between aligning mental model of users, their decision-making
and the structure of the visualization. It is very important for FSTs to be fast and precise in their job,
and service tooling is made for providing them with appropriate information at the right time. It has
to minimize the search and analysis times, which can be achieved through aligning their thinking
process and the structure of visualization. This relation was applied in the visualization designed and
the results are summed up in current thesis.
1.1 Context of use
The first step to take in developing data visualization is to understand who the user of that data
is and what kind of information he does need. Knowledge and background of the user, his time and
cost constraints, as well as his motivations and intentions have to be considered.
Noah Iliinsky and Julie Steele (2011) in their book “Designing Data Visualizations” proposed two sets
of questions to be answered before the one decides to create data visualization:

Context of use:
a. What information does my user need to be successful?
b. How much detail does he/she need?
6
c. How long does he/she have to make it effective?

Data:
a. How many dimensions does it have? Which are the most important ones?
b. What sort of relationships do they have?
These questions can help to define the background and describe the two most important aspects of
data visualization, data itself and the reader. This chapter will answer to all aforementioned
questions.
1.1.1 Who is Field Service Technician (FST) and what information does he need
to be successful?
The purpose of the visualization that has been developed in the process of this assignment is to help
Océ FST’s to perform their maintenance activities in a more efficient way. In order to be efficient, a
FST has to be able to retrieve and understand the data extracted from equipment as fast as possible,
and data has to be able to signal on all errors and warnings concerning product without being biased
and confusing. In order to achieve these goals, it has to be clearly defined which data has to be
extracted and shown to FST as information needed for maintenance.
As a first step, goals and needs of FST have to be explained. As defined by Océ:
“FST’s provide service in terms of system installation, corrective maintenance, preventive
maintenance, modification integration and operator training for the standard products.”
Tasks and activities of FST relevant to the current assignment include:
-
Preventive maintenance to ensure Océ products perform optimal efficiency;
-
Identification of problems and on-site repairs including tests;
-
Escalation of unresolved problems according to predefined service procedures;
After acquiring a clear view on tasks of FST, knowing what kind of data is stored within products
that could help FST to perform his activities is a next step in creating a successful visualization.
1.1.2 Data type
Products manufactured by Océ are spread all over the world. Every product accumulates multiple
sets of data on its performance. In order to be able to process this data from different products, type
of data as well as extraction procedure has to be unified.
Data that is being stored within Océ products has undergone multiple transformations for the
last 15 years.
First Océ products have stored data in OTS format, which later has been modified to SDS-Log. In
SDS-Log format, data is stored in Microsoft SQL database. Both of mentioned data types are still used
in some products that have been released to the field in the past, but the amount of products with
those data types is decreasing as new products are being manufactured with a different data type –
DataLog, which is a collection of all relevant product data for analyzing problems in an Océ product
7
or analysis of the usage of an Océ product. The format is standardized for all products that need
dedicated Océ service support: printers, scanners and software applications (servers).
The DataLog is maintained by the product and can be exchanged with the service laptop, sent to the
central archive which is accessible by OpCo-1 as well as HQ -staff or remotely retrieved. Below is an
overview of typical applications for which the DataLog is intended:
1. Pre analysis done at call screening or service department, with objectives to avoid service
visit where possible and prepare the visit (check presence of necessary parts, tools etc).
2. On site analysis by a FST. The service application, presently known as STAR, is capable of
retrieving the DataLog from the product and presenting the relevant data. To retrieve the
DataLog a communication has to be set up between a service device (presently a laptop) and
product (presently the controller).
3. Post analysis by service department or R&D. All DataLog files from products serviced by Océ
are transferred to a data warehouse. This data is useful for analyzing problems in certain
populations, or over products.
Data that has been accumulated by the product can be represented in two ways:
1. Snapshots. A snapshot represents all data on the system that is relevant and helpful in
problem analysis. Every snapshot is created in the end of a service visit. In order to limit
volume of data saved, amount of snapshots is restricted to 7.
2. Event data. Event data represent information on events that occurred within the system,
such as: errors, modifications, replacements.
How many dimensions does data have? Which are the most important ones?
Data extracted from Océ products is multidimensional. To get a full understanding of processes
ongoing inside the product, and being able to fix errors as well as perform preventive maintenance
activities, following data dimensions are being used:

Error code – is a name of error in database, which helps to retrieve description of an
appropriate error;

Description – describes an error;

Date, time – shows when did an error appear;

Counters – show absolute values of how many actions have been performed (how many
prints, cuts etc.);

Description of the counter – what kind of counter is used;

Absolute occurrence – how many times did an error occur;
These dimensions of data help to build an overview of what has happened, when, and how often
what results in an ability to draw certain conclusions and define patterns.
1
Operational Company. Océ branches over the world.
8
What sort of relations do they use?
Unfortunately the database that is used for STAR tooling has been developed some time ago, and
it turned out to be impossible to acquire class diagrams of it, and it would take substantial additional
time, time that I did not have, to define and describe relations between data. However, relations
between information presented in the tool’s interface, could be used to ‘reverse engineer’ and build
a new visualization.
1.2 What is data visualization?
Since the creation of a very first computer, one has been saving and accumulating more and
more data. Data on system performance, processed operations, and errors are saved constantly and
need a way to be presented to the human in a form of understandable information that could be
turned into knowledge, Table 1 (Chen M., 2009).
Table 1 “Russell Ackoff’s definitions of data, information, and knowledge in perceptual and cognitive space” (Chen
M., 2009)
Category
Definition
Data
Symbol
Information
Data that are processed to be useful, providing answers to
“who,” “what,” “where,” and “when” questions
Knowledge
Application of data and information, providing answers to
“how” questions
The need for visualization is based on the difficulties humans face in acquiring a sufficient
amount of information (Pinfo) or knowledge (Pknow) directly from a data set (Cdata). The process of
creating visualization is a function that maps from Cdata, with control parameters applied (Cctrl), to the
set of all imagery data, Cimage. It transforms a data set Cdata to a visual representation Cimage, which
facilitates a more efficient and effective cognitive process for acquiring Pinfo and Pknow (Chen M.,
2009).
Picture 1”Visualization process” (Chen M., 2009)
Any visual representation of data that fits criteria mentioned further can be referred to as data
visualization:
9
● Algorithmically drawn (may have custom touches but is largely rendered with the
help of computerized methods);
● Easy to regenerate with different data (the same form may be repurposed to
represent different data sets with similar dimensions or characteristics);
● Often aesthetically barren (data is not decorated);
● Relatively data-rich (large volumes of data are welcome and viable),
There are two general types of data visualization:
● Exploration – exploratory data visualizations are used when there is no particular
knowledge about the content of data, therefore data has to be analyzed with a purpose of
finding the story the data has to tell;
● Explanation – explanatory data visualizations are used when it is known already what
information does the data contain, but it has to be explained to other people who are not
familiar with the data (Iliinsky N., 2011).
In data visualization there are three basic entities, and the nature of visualization depends on
which relation between these participants is dominant. The participants and relationships between
them are reflected in the Picture 1 (Noah Illinsky, Julie Steele, 2011.).
Picture 2 “Data-Reader-Designer-Trinity” (Iliinsky N., 2011)
In case of Océ, an informative visualization would take place, as this is the kind of visualization
that helps to explain data to the reader, who is the FST. Informative type of visualization is aimed on
educating the reader and turning big volumes of data into relevant information for the user in a way
that it could be easy to understand. Test prototype of data visualization has been created, which was
tested among FST’s, to determine if it fits the requirements of FST and outperforms current tooling.
2 Conceptual Framework
2.1.1 Exploration of visualization strategies
To determine changes and new findings on visualization strategies since the introduction of
current strategies used by Océ, literature review has been performed. Literature on visualization
10
strategies depending on type of data, type of users and usage of colors, shapes, positioning as well as
algorithms made to optimize performance of software is widely available. To develop a visualization
that would help to improve maintenance activities, it was important to analyze current maintenance
process that involves current software. Assignment also involves development of advices on user
interface; therefore relevant literature has been studied as well.
2.1.2 Determining strategies for the given case
To define which visualization strategy would be most appropriate for the given case
interview with end users have been arranged. Qualitative, semi-structured interviews were
conducted. Semi-structured interviews imply that there is no set of particular questions but only an
interview guideline, which outlines topics to be covered, and interviewee has a freedom on how to
reply.
Purpose of interviews was to get a “natural” input from FSTs about their concerns on current
visualization strategy to define key requirements for the future visualization as seen by FSTs and
accordingly to their feedback and literature study create a most appropriate visualization.
The limitation that should be considered when interviewing FSTs is that they are used to the
current interface as it is present for the last 15+ years. By nature, people are somewhat resistant to
use new different interfaces (Davis F. D., 1989). After prolonged usage of an old interface, they might
not see new hidden opportunities.
2.1.3 Test of chosen strategy
In order to determine if visualization strategy is suitable for the type of data that is being
extracted from Océ products and if it enables FST to perform their maintenance more effectively and
efficiently, a digital prototype was created and tested among FSTs. The prototype was rated on
multiple criteria, using the expanded SUS evaluation model (Brooke J., 1996).
Due to limitations in time and availability of FST’s only a single test is possible with 5 FSTs and we
have to fully rely on their opinions about the prototype. Single testing with a very limited amount of
participants excludes quantitative measurements as they would not provide reliable enough data.
Comparison with the old tooling would also not provide valid results, as it requires some time for
users to get acknowledged with new tooling for fare comparison of it against old tooling.
2.2 Decision-making
In the case of Océ FST’s, decisions are made in bounded rationality context, which is a narrowed
theory coming from Savage’s Paradigm. There are 3 essential building blocks of the Savage’s
Paradigm theory:
1. A set of alternative states of the world which are beyond the decision-makers control;
2. Set of alternative actions available for decision-maker, or just acts;
3. A set of alternative consequences (Radner R., 2000).
In addition to the 3 conditions mentioned every FST has his own beliefs on what should be his act
based on state of the world and to which consequences might it bring.
11
That leads to a wealth of possible interpretations and potential complexity. Bounded Rationality
theory developed by Herbert Simon implies that Savage’s Paradigm is too broad, and introduces
limitations of actor’s ability to process information and his knowledge (Simon A. H., 1991).
When FST is performing service visit his goal set by the company is to eliminate all issues that
cause or potentially can cause malfunction of the product. To enable FST to do so, over 15 years ago
Océ has developed a service tooling that visualizes data accumulated within the product. But besides
that data there are also comments of the customer, environmental factors that might influence the
condition of media used and quality of prints. Using current tooling FST is absolutely free in applying
his own methodology in performing maintenance. He is free to make decisions upon his knowledge
solely, without using service tooling, using a certain part of tooling, comments of the customer or any
other combination of aforementioned sources, considering the fact current tooling has no structure
and consists of multiple separate applications (more details on current tooling can be found in
Appendix 1).
As proved by researches conducted in recent years, a world-class expert in any field can hold in
his memory about 50,000 familiar units of relevant information. This body of knowledge is stored as
an encyclopedia in the brain of an expert and with correct triggers that information can be called out
and used by the expert (Simon A. H., 1991). FST is not able to take into consideration all information
provided by the service tooling, customer and environment at once, so it is important to highlight the
most relevant and important information at the given moment and trigger that body of knowledge
stored in the brain of FST. Application of amount and types of sources depends on how quickly FST
can find solution that responds to his belief on what the reason of the issue is. As soon as relevant
possible cause is found, he applies the corresponding technique to solve the issue. If his applied
technique solved the issue, then FST stops looking further and proceeds to his next task if there is
any or closes the job. This method is called satisficing that falls under one of bounded rationality
theory developments, and it implies that decision maker has a certain aspiration as how good the
alternative option should be, and as soon as he finds a good enough alternative he terminates the
search and applies that option. If his chosen option does not solve the issue he moves to the next
one etc. Unfortunately current approach is not the best for the given case. Due to specifics of the
maintenance performed by the FST, the first option that fits beliefs of the FST may not be the right
one even if it temporarily solves the problem, what can result in occurrence of a problem again and
require additional service visit that results in extra costs and dissatisfaction of the customer.
Before applying any solution and drawing conclusions, FST is expected to gather and analyze all
possible information. In depth analysis might highlight a root cause of the problem, whereas the first
issue that fits beliefs of the FST might be only a consequence of the root cause. The challenge was to
develop new visualization in such way so it would change the decision-making process from
satisficing to the search of optimal solution, which requires analysis of most of available information
that could trigger the body of knowledge in the brain of FST and help solving the root cause. The
challenge of how to create a transition from satisficing method, which implied different maintenance
approach from every FST, to approach that implies search of an optimal solution has been solved
12
through implementation of the concept used for training FST’s into the visualization. The concept is
described in the section below.
2.2.1 CARE concept
CARE concept is a methodology developed by Océ with a purpose to teach all FSTs a single
approach on how to perform their maintenance. Every FST undergoes training that takes place every
few years where he is taught improved service methods and enhances his knowledge of the product.
CARE concept is a core of FST service visit. It describes steps that he is supposed to take from a very
first second of a service visit up to the moment he leaves the customer’s site. The concept consists of
the following steps:
1. Collect. On this stage FST is supposed to collect all possible information that he can: ask
customer to describe the problem as detailed as possible, try to reproduce the problem,
observe the environment, study information provided by the system;
2. Analyze. This step implies usage of additional service tools for analysis. FST has to look into
error history, previous service visits, technical manuals etc. to get as much relevant
information on possible cause of the problem and problem itself as possible in order to be
able to solve the problem properly and try to exclude the chance of its occurrence again in
the nearest future;
3. Repair. This is the actual physical reparation of the product. Where necessary FST uses
appropriate tools and parts to fix the issue.
4. Evaluate. After product is repaired and FST made sure that issue does not persist and
customer is not complaining anymore, FST has to fill out his service report, where he enters
all actions that he has taken, parts that he has used etc.
Current concept is supposed to teach all technicians the same working approach, standardize
their working method by aligning it to the same high quality requirements.
2.2.2 CARE concept and decision-making
Even though every FST has been trained to apply CARE concept in their service visits, based on results
of interviews it is not being done. Every FST has its own approach on performing maintenance
activities, consequences of which are described in a section above.
On initial stage of developing visualization main emphasis was on presentation of information, its
layering, structure, addition of graphical representation etc. But on the further stage after 1st round
of interview, when Océ management showed its intention on bringing all FSTs to the same approach
in performing service visit it has been decided to apply CARE concept to the visualization. As
mentioned before, there are 4 steps FST has to take:
1. The first step is collecting all information available, without making any decisions.
2. Analyze the available information using analysis tools, such as error overview over time,
values of consumables, history of previous visits and available graphics.
13
3. Draw certain conclusions on what might be the possible cause of the issue and perform acts
that relate to his beliefs based on collected information about the world if I use terms
proposed by Radner (Radner R., 2000).
4. Fill out evaluation part, where he reflects on actions taken and resources used.
This approach is imposed by the new visualization strategy. Structure of visualization consists of four
aforementioned CARE steps, whereas first step is home screen that is shown to FST, and Analysis and
Reair&Evaluate are 2 other items in main menu that FST can switch among. The application of the
maintenance approach taught to technicians, which is supposed to be a mental model of the every
technician when performing maintenance, to the data visualization was hoped to help to switch their
decision-making approach from satisficing, which is not wanted by Océ and can bring additional
costs, to the search of optimal solution, that provides a proper, long-term solution of the problem, as
it implies analysis of all available information and elimination of root problem. More detailed
description of visualization can be found in Appendix 3. Description of the prototype.
3 Literature review
Designing and testing of user interfaces are very well studied areas which allow making design
choices that have been tested and proven to being effective by multiple researches in different
areas. As mentioned in the book by Iliinksy and Steele (2011) there are many different types and
approaches for creating visualizations, so it is vital to set purposes for your visualization as well as
realize who is going to be the user of the visualization and what is the data to be displayed. Sources
like Data Visualizations by Iliinsky as well as “Data, Information, and Knowledge in Visualization”
(Chen M., 2009)provide explicit and in-depth knowledge on creating visualization strategies
depending on types of data, users and purposes.
After creating the visualization the next logical step is to test whether it represents any value and is
better than the previous versions. Many authors provide researches on user interface testing
techniques, highlighting advantages and disadvantages of a certain technique. After studying such
publications ( (John E.B., 1996), (Karat C., 1992), (Jeffries R., 1991), (Nielsen J., 1994). (Rowley E. D.,
1992), (Pintelon L.M., 1992), (Lewis C., 1990)), empirical methods have been chosen, namely
cognitive walkthrough in combination with think-aloud protocol. Cognitive walkthrough has been
described as one of most efficient and effective methods that do not require high volume of
resources and can be reliable even with smaller amount of tests ( (Karat C., 1992), (Jeffries R., 1991),
(Nielsen J., 1994)). Think-aloud protocol in its turn allows following the thinking process of the user
and notice when mental model of the user is not any further supported by the visualization which
results in difficulties for user to find and use information he requires.
The last step of this research was to evaluate the prototype, translate results of testing into certain
system that would allow understanding whether goals were achieved. The evaluation model that has
been chosen was SUS, which has been described in literature as a reliable model for evaluating user
interfaces ( (Lewis R.J., 2009), (Brooke J., 1996), (Bouvier D., 2012)). However, after proper analysis
of the SUS and with consulting Océ Design user interface testing specialist it has been decided that
14
statements chosen to evaluate criteria do not cover all the relevant aspects. For FST it is very
important to acquire substantial amount of information to perform necessary tasks, but there also
should not be too much of information (Smith L.S., 1986). Necessity to know whether information
provided is sufficient or not is reflected in the first (A, B) statement added to the SUS model. SUS
model was not created to evaluate the relation between visualization and the decision-making that
was the key factor to be evaluated in assignment. That evaluation is done through inclusion of
additional statements 2, 3 and 4 below. Statement 5 is designed to provide overall evaluation of the
proposal. All statements that were added to the original SUS model can be found below:
1. A) The information provided was not sufficient to perform my tasks;
B) There is too much of information which makes it difficult to highlight relevant information
necessary to fulfill the task - statements aimed on providing feedback on quality of
information displayed;
2. The implication of the CARE concept into the application helps me to navigate more easily –
statement aimed on providing feedback on the usefulness of implication of CARE concept;
3. A) The action I wanted to take next was always easy accessible;
B) I got a feeling as actions I wanted to take next were predicted by the tooling, so it was
always one click away – statements aimed on providing feedback on how well interface is
aligned with user’s mental model;
4. It was easy to make decisions using this tooling – statement aimed on providing feedback on
how well interface supports user in decision-making;
5. I see the prototype as an improvement – statement aimed on overall evaluation of the
tooling.
Another dimension that is vital for FSTs to be able to perform well is having data visualized in such
way that it gives quick and informative view over data, but not all data at any given moment, but
certain data in certain moment of service visit. Every FST is being taught to perform service visit
within a certain framework, following certain steps. Current tooling has no connection, alignment or
whatsoever with the concept FSTs have to follow while performing maintenance visit. Making
visualization aligned with a service visit framework (which also becomes a mental model of FST after
certain number of trainings and service visits) was one of the purposes of the visualization designed.
FSTs have to make decisions based on the information presented, so there is direct connection
between visualization and decision-making. Surprisingly, not much of research has been done in the
area of relation between visualizations and decision-making. Studying resources ( (Engers T. M.,
2001), (Hahn J., 2000), (Pintelon L.M., 1992), (Tsang A. H.C., 1995), (Simon A. H., 1979), (Simon A.
H., 1991), (Radner R., 2000)) allowed me to understand that visualization plays a very important role
on how decisions are made and why. This issue is extremely important in the case of this assignment
as FSTs are fully relying on information provided in the visualization designed. All decisions they
make are based on information presented and their productivity also heavily depends on the quality
of presentation.
15
In this research I have observed how aligning visualization with mental model of user (service
concept they are taught and required to apply) helps users to perform their tasks better and increase
their productivity.
4 Interviews
4.1 Approach
To understand the needs and requirements of FSTs in relation to tooling they use I have conducted
multiple interviews. It has been decided to conduct 3 rounds of interviews with FSTs responsible for
maintenance of 2 different products: document printing (DP) and wide format printing systems
(WFPS). From every business unit 3 FSTs have been chosen on the basis of following criteria:

I was interested in best performing, most reliable FSTs with a good track record. The reason
behind this requirement is that I wanted to deal with FSTs who utilize all capabilities of
current tooling and can give its complete and objective overview;

I was also interested in FSTs with different experience level. My hypothesis was that FSTs
who are employed by Océ in their current position for a period of time over 10 years,
experts, would have different opinion about current tooling than FSTs employed less than 10
years. The reason for choosing 10 years of experience is that as mentioned by Simon it is
impossible to become an expert in an area of specialty without dedicating at least 10 years to
learning and practice (Simon A. H., 1991). Less experienced FSTs, who turned out to be also
younger, could have more open view on the tooling and be able to see capabilities that
tooling is lacking. More experienced employees are less dependent on the tooling and more
used to it, what makes it harder for them to detect flaws and shortcomings of current
tooling. On the other hand being well acknowledged with capabilities of the current tooling
with a presence of good analytical skills they might be able to provide valuable in depth
opinion over tooling.
These criteria have been provided to the service managers and they proposed FSTs that have fulfilled
requirements mentioned above.
The reasoning behind decision to conduct three rounds of interviews was as follows:

1st round – gathering information. Qualitative, semi-structured interviews were conducted.
Semi-structured interviews imply that there is no set of particular questions but only an
interview guideline, which outlines topics to be covered, and interviewee has a freedom on
how to reply. So FSTs had absolute freedom on expressing their thoughts, concerns and
opinions over current tooling. List of topics that were discussed and summary of results can
be found in Appendix 2.

2nd round – first draft of prototype and summary of proposals created were shown to FSTs
with a purpose to acquire their opinion over ideas developed and directions taken.
Qualitative, semi-structured interviews.
16

3rd round – test of final version of visualization. FSTs were provided with final prototype and
test was performed with a goal of defining whether developed prototype raises productivity
of FSTs.
4.2 Results
1st round of interviews provided a range of insightful and very different opinions over current STAR
tooling version. Many issues were mentioned that concerned every aspect of STAR tooling. After
processing results of all interviews I have made a summary of concerns that were mentioned most
and were most desired and significant improvements. The list of proposals was used for determining
what data should be used and converted into information and how should it be layered.
2nd round of interviews was used for validation of the directions I’ve taken. I have presented first
draft of the prototype and described reasoning of the design choices I’ve made with a request to
comment and express their opinions over every element. Every FST expressed his concerns about not
being able to test the tooling before commenting on it, but they were explained that testing and
evaluation were planned for the 3rd meeting, but purpose of current meeting was to validate if right
issues were tackled. Overall feedback was positive, the directions taken were confirmed and FSTs
expressed high interest, excitement in the tooling and appreciated design choices made.
3rd meeting round was organized with a purpose to test the Power Point prototype developed and
ask FSTs to solve a real customer complaint that has been solved remotely without having Océ
product at hand. While solving the issue FSTs filled out the issued form with a set of questions and
SUS evaluation model. Every FST rated prototype positively and assigned his own ranking accordingly
to the SUS model. The final score for the prototype reached 75%, where any result over 60% is
considered as a good result (Brooke J., 1996). Results of the filled out forms can be found in the
section Results.
More detailed description of all rounds of interviews can be found in Appendix 2.
5 Design of new visualization strategy
As mentioned in previous section, design phase started with acquisition of feedback from direct
users (FSTs) which was used as a foundation for the building of new visualization strategy. Comments
acquired from FSTs have been summarized into proposals that can be found in Appendix 2. That
summary reflects needs of FSTs in current environment that current tooling can’t fulfill.
While conducting interviews I have faced multiple opinions which at certain points could be
described as controversial an excluding each other. To be able to make correct design choices which
would maximize satisfaction of opposite parties simultaneously being an effective and efficient
solution, I have adopted rules (Smith, Mosier. 1986) that helped me to make scientifically based
decisions.
17
The first and most significant proposal was creation of unified dashboard, where all data that
currently is spread amongst multiple separate applications would be presented in one unified
dashboard. The challenge was to choose all the data that might be required by FSTs to perform their
tasks and present it in a way that would satisfy FSTs as well as product specialists as these two
parties had quite different views on what kind of data should be presented and how it should be
transformed into information. Having this task to solve, I have applied recommendations from the
paper of Smith and Mosier which are as follows:

Necessary Data Displayed. After studying current tooling, having 2 rounds of interviews and
consultation with Field Service Specialists it has been determined which data should be
presented to give FSTs opportunity to get the full picture of the product’s performance;

Only Necessary Data Displayed. On the other hand there should be no overflow of data. Too
much of information can mislead and confuse FSTs during their maintenance activities. It will
also take more time to scan through the information presented with a purpose of finding the
appropriate.

Data Displayed in Usable Form. Users should be able to use data directly, without necessity
to convert or transform displayed data.

Data Display Consistent with User Conventions. The current version of STAR tooling has been
used for over 15 years and most of FSTs’ interviewed have used it for the whole period which
means they are used to the way data is presented in the current tooling. It was important to
stay consistent with the way data was presented in current tooling so that when FSTs face
new version they would be able to understand and easily navigate through the information.
For the same purpose, to help FSTs to make decisions and navigate, CARE concept has been
implemented in the design of prototype.

User Control of Data Display. It is important to allow users to control the amount and
complexity of data. Current issue has also been mentioned by FSTs themselves, as depending
on their experience and habits, they might require different amount and types of data. The
ability to control amount and complexity of data, has been provided through implementation
of block design that allowed users to prioritize and show/hide data they desire.

User Changes to Displayed Data. The possibility for user to edit data presented is vital in the
case of current visualization due to the fact it is very important to record all information
provided by the customer, all activities performed and remarks for future maintenance.

Context for Displayed Data. User should not be pushed to remember the meaning and
purpose of data presented. With this purpose the for every data segment presented a
context should be set up. In the case of current visualization, every new data segment in
visualization has concise description accompanying or a link to further explanation.

Familiar Wording. All item naming has been kept from the current tooling. This helps users to
navigate easier and better understand additional information provided by the prototype.

Consistent Wording. Consistency in wording and presentation methods is one of core
requirements of good visualization. In the case of current visualization the same labels are
used for the same types of data throughout the visualization. Same techniques are used for
18
presenting graphical data. Consistency, or memorability, as proposed by Océ Design usability
testing specialist, is used as one of the core criteria in evaluation of the visualization.
List above was used as general guidelines for the design of visualization that helped to incorporate
most of the desirable components and remarks of FSTs as well as product specialists. Many of the
statements listed above were not taken in account when developing tooling used now which puts it
in disadvantageous position in relation to the proposal described in current research.
One of major flaws of current tooling was lack of graphical representation. This issue was heavily
stressed by FSTs as a more efficient and effective way of showing trends and overall behavior of the
product.
There are multiple ways of graphical representation of data, in the case of current visualization, due
to the nature of data it has been decided to choose between linear and bar graphs. The data that was
chosen for graphical representation is time series data, which implies time-value relation. After
consultation with two representatives of Océ Design and literature (Iliinsky N., 2011), it has been
decided to choose bar graphs due to the fact that linear graphics imply relation between values
which is not the case in current visualization. Bar graphs do not imply logical relation between values,
but still allow seeing the trend in occurrence of a certain value which plays decisive role in deciding
whether there is a problem that requires attention of the FST or it is just an accident.
For being able to evaluate whether design choices made are able to provide Océ FSTs with a better
service tooling that they have now it was decided to develop a prototype which would allow testing
it and acquiring certain results.
There are multiple testing methods for user interfaces, and initial choice fell upon paper prototyping
or PowerPoint prototyping. Developing real application, even with limited capabilities, would require
time and resources that exceeded given constraints.
Paper prototyping does not provide enough of interactivity so it was decided to develop PowerPoint
based prototype with making it as close as possible to “real”, working application. PowerPoint
provides enough capabilities to build a user interface with ability to create buttons and visual
materials, which was enough for building a prototype. More detailed description of the prototype
can be found in Appendix 3. Description of the prototype.
6 Test and evaluation
In order to determine whether proposal developed is able to achieve goal set before assignment
have started, namely raise productivity of FSTs, testing and evaluation had to be performed.
There are multiple UI testing techniques available. In this particular case an empirical testing method,
cognitive walkthrough, in combination with think-aloud protocol has been applied.
Cognitive walkthrough implies a group of users performing a certain task using interface what allows
to highlight problems within the interface. In given case a group of FSTs were given a task - to solve
19
the real world problem using the new tooling’s prototype which allowed detecting most important
issues.
Think aloud protocol methodology implies user of interface describing aloud every action he takes
and thought he has that are further analyzed or recorded and analyzed afterwards. Think aloud
protocol allows following the user and defining for what extent is the interface aligned with user’s
mental model. In given case every FST is being taught to work within CARE concept and tooling is
developed taking in consideration the same concept, so it has been expected that FST would be able
to easily navigate through the tooling.
The task given for FSTs is described in Appendix 4. Description of task for a test.
Except of SUS every FST was asked to answer 3 questions:
1. Describe actions taken and write down all relevant information;
2. Describe what did he like/dislike about the proposal and what would he improve;
3. Did the alignment of visualization and CARE concept help?
With answering those questions FST allowed us to better understand how well was the prototype
designed.
For being able to decide how successfully has the test been and being able to analyze its results an
evaluation model has been adopted.
System Usability Scale (SUS), developed by Digital Equipment Corporation in 1986 still remains as one
of the most used models with a high reliability coefficient to evaluate user interface (Lewis, Sauro.
2009). This end-of-test model allows sufficient subjective evaluation of the interface by its users.
Even though SUS covers most of criteria that describe the interface in particular given case a
necessity of slight modification of the scale has been required. After consulting literature and Océ
Design usability specialists it has been decided to add additional statements described in section
Literature Review. The whole scale with additional statements can be seen below:
Tabel 1 “System Usability Scale”
Criteria/Measurement
Strongly
Disagree
Agree
Not Sure
Disagree
Strongly
Disagree
1. I think that I would like
to use this system
frequently
2. I found the system
unnecessarily complex
3. I thought the system was
easy to use
20
4. I think that I would need
the support of a
technical person to be
able to use this system
5. I found the various
functions in this system
were well integrated
6. I thought there was too
much inconsistency in
this system
7. I would imagine that
most people would learn
to use this system very
quickly
8. I found the system very
awkward to use
9. I felt very confident using
the system
10. I needed to learn a lot of
things before I could get
going with this system
11. The information
provided was not
sufficient to perform my
tasks
12. There is too much of
information that makes
it difficult to highlight
relevant information
necessary to fulfill the
task
13. The implication of CARE
concept into application
helps me to navigate
more easily
14. The action I wanted to
take next was always
easy accessible
15. I got a feeling as actions I
wanted to take next
were predicted by the
21
tooling, so it was always
one click away
16. It was easy to make
decisions using this
tooling
17. I see the prototype as an
improvement
Each of statements is corresponding to a certain criterion, numbers 1, 2, 3, 5, 7, 8, 9, 11, 12
correspond to Usability; numbers 4 and 10 to Learnability; and number 6 to Memorability.
Definitions of all criteria can be found in Appendix 5. Reasoning behind statements 13 to 17 was
described before.
Every criterion is measured on 5 point scale, 5 being anchored as “Strongly agree” and 1 “Strongly
disagree”. To calculate the SUS score, first the score contributions from each item were summed.
Each item's score contribution ranges from 0 to 4. For items 1, 3, 5, 7, and 9 the score contribution is
the scale position minus 1. For items 2,4,6,8 and 10, the contribution is 5 minus the scale position.
To obtain the overall value of SUS the sum of the scores was multiplied by 2.5.(Brooke, 1992).
SUS scores have a range of 0 to 170. For every measurement graded below 3 a comment has to be
provided that will state why this item scored negative grade.
There are two additional criteria that are important for drawing conclusions on how successful the
prototype is: efficiency and effectiveness. Efficiency could be measured through the time invested in
solving a task provided to FSTs, but due to the very small sample and disability to run multiple tests
with every FST it has been decided that quantitative values would not be reliable enough and
efficiency would not be measured at this time.
Effectiveness of the prototype has been measured through requesting FST to write down all relevant
information for solving the problem. Depending on how much relevant information was found and
comments of FST it has been determined whether prototype responds to expectations.
22
6.1 SWOT of new strategy
SWOT analysis is used for defining strong and weak aspects of the object and trying to find
opportunities on how to utilize those strong aspects and minimize weaknesses as well as avoid
threats and use opportunities. Below SWOT model for the developed visualization strategy can be
found:
Tabel 2 “SWOT”
Strengths
Weaknesses
Unified dashboard
Learning curve
Graphical representation of data
Internet coverage dependable
Consistency with CARE and current workflow
framework
Web Based
Multi-layered
Decreased cognitive load
Time efficient
Supports media upload
Opportunities
Threats
Expand with new information, such as
User resistance
parameters
Compatibility with Canon products/strategies
Adjustability
Precision of run out dates estimation
Multi platform
Strengths VS Threats:
User Resistance VS Consistency with CARE and current workflow: With new version of STAR tooling
the information that is provided to FSTs wasn’t changed much itself, FST can find same sources he
used before, but access them faster and easier. Also FSTs are being taught to work following CARE
concept which is also integrated in the new tooling, what allows FSTs to intuitively navigate through
the tooling following CARE concepts rules. That being said, consistency of new tooling with CARE
concept and type of information provided allows to expect low user resistance rate.
Compatibility with Canon products/strategies VS Adjustability: New tooling is web interface based.
This allows it to be easy adjustable; with data present in database it can be easily retrieved and
displayed. Information can be relocated, moved and modified with minimal effort depending on
requirements.
23
Opportunities VS Weaknesses:
Internet coverage dependable/Multi platform: As tooling might be used on the Smartphone, in case
of absence of Wi-Fi connection, FST might use mobile connection on his Smartphone (3G, 4G). In case
of absence of any internet connection, FST would have to rely on local database.
Strengths/Opportunities:
Expand with new information, such as parameters, adjustable/Web based adjustability: During
current assignment a way on how to provide FSTs and back office with more of important
information have been discovered. Currently there are no parameters registered and presented in
relation to a certain error. This option would allow performing better analysis on the reasons of
errors and exploring patterns in product behavior. Due to tooling being based on web interface, it is
highly adjustable and new information can be added to the current setting.
Multi platform/web based: Due to the tooling being web based it might be used on multiple
platforms, such as product’s LCD display, Smartphone, tablet etc.
Weaknesses/Threats:
With alignment of the visualization and CARE concept taught to FSTs the learning curve of the
product was minimized and allowed FSTs to navigate through the visualization easier, which is
expected to decrease user resistance, as it will be easier for them to learn how to use the new
tooling.
New visualization strategy appears to have enough strong points to minimize weaknesses and avoid
threats through utilizing opportunities.
7 Results
After running 1st round of interviews FSTs defined their view on what should be changed and/or
improved. Taking in consideration comments from FSTs and personal investigation I have built a first
draft of prototype that included those observations. The core desirable features mentioned by FSTs
were unified dashboard where all current application would be integrated in one place and
implementation of graphical representation of data.
On 2nd meetings with FSTs they were asked to reflect on the proposal I have prepared. Overall
feedback was positive and considering remarks acquired after meeting with technicians and
specialists from HQ I have proceeded to further development.
When the prototype was ready a 3rd round of meeting was arranged where FSTs were asked to test
the prototype.
Due to the limited availability of FSTs as well as time constraints of the internship it was only possible
to arrange 4 subjects, all FSTs, to test my prototypes. Every FST attempted to solve the given problem
24
using the prototype and afterwards using SUS model and open questions evaluated his experience
with the prototype.
FST number 1 was able to solve the problem using the prototype without any visible difficulties, but
mentioned that due to the fact he is unfamiliar with the product used for the test and because he is
not a full-time FST at the moment, it was quite difficult to perform proper job and that resulted in 7
neutral responses, “Not Sure”. Overall prototype was described as a significant improvement and
evaluated for 65% which is just over the minimal score that allows considering user interface as
successful. He could easily navigate through the system and find all the information necessary for
solving the problem. When asked on what does he find as most significant improvement and
disadvantage of the system it was mentioned that graphical presentation is a highly appreciated addon and alignment with CARE concept helps to navigate.
FST 2 also solved the issue without any extra effort and could navigate through the system with just
brief explanation of the core functionality. All necessary information has been discovered and a right
solution was proposed. FST rated prototype for overall of 75%. When asked about what would he
improve if any and was liked the most, FST responded that presentation of information is designed
very well, but before drawing any conclusions he has to use it in the field. It was mentioned that
graphical representation gives a quick idea on the condition of the product and presence of any
trends.
FST 3 rated product for 85% which is a highest rating amongst all and provided most detailed
description of every step taken and most explicit feedback on prototype. FST 3 also managed to solve
the issue without external help, but expressed concerns about the structure of the graphics. He has
found it difficult to follow the sequence of which graph follows which and felt lost when digging
deeper into analysis using graphs. It was advised to introduce a map of graphs similar to the one used
for website maps. When user is viewing certain graph and from it proceeds to another graph, it
should be shown where is he located now and how can he get back or to another appropriate graph.
Another concern expressed was whether there is too much information displayed now, but further it
was decided that this impression came due to the fact prototype is seen for the first time and is very
different from the current tooling and provides much more information.
FST number 4 expressed most excitement on the graphical representation of the information. Due to
the fact he was not familiar to the type of the product used, he also could not fully reflect on the
prototype, but even without knowing the product, using his common knowledge as FST and structure
of the prototype he has managed to find a solution for the problem provided. Prototype was rated
for 75%. He stressed the usefulness of the alignment of CARE concept and the visualization and
noted that it makes it easier to navigate through it and abide the concept taught without jumping to
conclusions too fast.
Overall prototype was rated at 75% which is considered as a good result and every FST expressed
desire to use it in the field with a purpose of full, real-world testing. Even though results of a test
25
including only 4 subjects cannot be considered as reliable enough, it gives an indication that
prototype has been developed well enough to be considered for further development.
Throughout the development period multiple meetings and presentations were organized with
people from multiple departments including product specialists, FSTs, developers. The directions
taken before the start of development and the prototype itself was approved by representatives of
all departments.
8 Conclusions
The following tasks were set before the beginning of the assignment:
Develop visualization strategy for creating and presenting the relevant information to the FST for
both corrective, as well as preventive maintenance activities. Develop proposals for visualization
strategy representing data extracted from Océ equipment that could help Océ’s FSTs to improve
their maintenance activities.
A secondary assignment was to identify additional data that would be beneficial for FSTs if it would
be available.
As of primary assignment, the visualization that has been developed got positive feedback from FSTs
and representatives of other departments who had a chance to see it, but due to the fact that
amount of tests was low there is no available conclusion with enough statistical proof. However,
results of tests that were performed are promising and the proposal developed will be used as a
starting point for further development.
I have used multiple sources to define the directions and make design choices when building
visualization with core criteria being:
1. Displaying only the relevant information;
2. Making it easy and quick to navigate and extract necessary information for maintenance;
3. Avoiding FSTs to make quick decisions before analyzing all the relevant information.
All those factors have been implemented in the visualization. All FSTs participated in development
and evaluation of the visualization believed that it is significant improvement in comparison to the
current tooling they use. It provides more information while staying simple and understandable in
use. By aligning it with CARE concept two issues were solved:

Learning curve has been minimized. Visualization is structured using same steps as described
in CARE concept which allows FSTs to predict what information every part of visualization is
holding without previous experience in using it.

Currently most of FSTs use satisficing approach to make their decisions which is not
acceptable for the company. Another goal of the visualization was to change that approach
from satisficing to the search of optimal solution. CARE concept implies collection and
analysis of all relevant information before making any decisions and with building new
26
visualization considering CARE concept this approach is taken one step further and it makes
it less likely that FST will jump to conclusion before collecting and analyzing the available
information. Through application of mental model of the user to the visualization I
attempted to change decision-making approach of FSTs. Whether this attempt was
successful or not can’t be judged now due to the small testing sample and time constraints of
the internship.
Within the context of current research I have attempted on changing the working approach of FSTs
from satisficing method to the search of optimal solution. Satisficing approach, which might be a
valid approach in some cases, is not desired by Océ. Quick search for a first solution that fits the FST
beliefs might result in customer dissatisfaction and increased costs, whereas search of optimal
solution implies higher chance of finding the root problem and solving it that significantly decreases
the chance of reoccurrence of the problem.
Current issue might be faced by many companies and also customers that are not satisfied with the
maintenance results in case problem occurs over and over again. Every company that provides after
sale service has a certain service approach that they apply and bringing all employees under the
same working method is desired. Providing trainings for service employees once in a certain period is
a necessary procedure but it does not guarantee that method taught will be applied, whereas
imposing that method in the service tooling used on every visit might be a more effective approach.
Knowing the general ideas of the approach from the training, service employees would recognize
them imposed in the visualization and unintentionally adopt that approach as it would be much
easier to perform maintenance using that method, which is aligned with the methods taught, then to
work using their own method which is contradictory with the methods taught and the structure of
the visualization. Alignment of visualization might also lead to improvement in the productivity, as
with more structured approach service employees would be able to contribute more time to a
problem solving itself instead of searching for appropriate information.
The clients that own products that require periodical maintenance are not interested in their
equipment being dysfunctional. They are interested in well performing products with as least as
possible maintenance and disorder time, as it costs time and possibly money. Visualization
developed within assignment is aimed on providing FSTs with a most relevant information at the
most appropriate time, which should be achieved through the alignment of their mental models and
the structure of the visualization. Users of the products are interested in this approach to be
successful as it will guarantee more reliable performance of the products and less required service
visits.
As discovered in Océ, service tooling is mostly being used as a practical tool for assisting on
performing required tasks. But it can also be used as an influence channel with a purpose to impose
certain ideas and working methods. Many subjects interviewed showed skeptical attitude to the
service approach they are being taught, but all of them agreed it helps having the same approach
being integrated in the service tooling, of course taking for granted that service method mentioned is
logic and well designed.
27
Current case study is also aimed on serving as a contribution to the bounded rationality theory. The
theory includes both decision-making methods described in the assignment: satisficing and search
for an optimal solution. While studying literature on the bounded rationality theory I have come
along explicit descriptions of both methods but did not discover any description of the
interchangeability of those methods. Current research might be used as a contribution to that area.
With current research I attempted on influencing FSTs to migrate from satisficing method that is not
desired by the company to the search of optimal solution using the same approach for all FSTs
instead of individual methods. I attempted to reach this goal through aligning structure of the
visualization and the mental model of the users.
The targets set before the beginning of assignment have been achieved to a certain extent, due to
the limitations in time.
9 Future Work
As mentioned in the Conslusions section there hasn’t been enough time to run sufficient amount of
tests of the prototype to draw reliable conclusions. In the future Océ should certainly run more tests
with a purpose to gather more results.
To change the working approach of FSTs is a long term goal, Océ is required to perform constant
monitoring and analysis to be able to tell whether visualization helps in achieving goals set.
Further investigation on interchangeability of decision-making methods mentioned in the assignment
is required. Those methods are explicitly described in the literature but it is not mentioned on how to
move from one method to another when necessary. One possible approach is mentioned within
assignment, but there are no reliable test results to prove it is valid. Also it might be useful and
interesting to do a research on interoperability of both decision-making methods. It might be the
case that for achieving a single task, for instance performing a service visit, both methods might be
applied depending on the specificity of the given case. For example, if it is required to restore system
as fast as possible to perform an urgent task, disregarding the long-term consequences a satisficing
method might be applied, which implies quick search for a first available solution, but has a high
chance of reoccurrence of the problem in the nearest future. But for the most it is more desired by
the company as well as the customer to perform maintenance that will result in stable and reliable
performance of the machine over a longer period. To achieve this search of the optimal solution has
to be performed. So complete exclusion of any of methods described above might not be the best
option. Application of combination of both methods might be most effective solution. A further
research on how to integrate both satisficing and search of optimal solution methods in the
visualization and aligning it with the mental model of the user might provide valuable results.
While working on the assignment some ideas that occurred could not be implemented due to the
limitation in time and knowledge. Some of those ideas were appreciated by Océ and are included in
this section.
28
Inclusion of additional parameters and data on environment
Parameters might be a helpful tool for finding the root causes of the occurred error. Customers tend
to use different media (paper, toner) from the media advised for use by Océ, which might lead to
paper jams and other issues that disturb the work of the product. Currently it is a very limited
amount of parameters that is stored within the product. Some information on type of media used
can be retrieved, but often it is not enough. FSTs need to have access to complete information on
what type of media has been used on specific days, in order to being able to draw conclusions about
the reasons of an error.
There might be two possible types of parameters:
A. Parameters coupled with errors: this type of parameters implies that parameters are logged
only when an error occurs. These parameters would not be logged continuously. Usage of
this type of parameters has its limitation but can be very useful for acquiring a better
overview of the state of the product on the moment of occurrence of an error. These
parameters might include media type as mentioned above (paper, toner), temperature,
humidity etc.
Having a limited amount of parameters it is possible to look for correlations between those
parameters and occurrence of the error. Not every relation might be noticed by the human,
especially if two factors are influencing each other, but the visible result occurs only after a
long period. In such case an automated search for correlation might be a valuable addition. If
data on parameters is available for a longer period, then it is possible to run an automated
search for correlation that might result in new patterns and new relations that might explain
occurrence of certain errors.
B. Continuous parameters. These parameters would be logged constantly. Continuous
parameters would give an opportunity to view the state of those parameters at any point in
time. For instance, raise of temperature and then decrease of it might cause condensation in
certain parts of the product that can in its turn cause malfunction. At the moment of service
visit and at the moment of occurrence of an error, temperature might be within acceptable
values. Logging temperature values on constant basis might help to view a graph of
temperature fluctuations and discover the values prior to error occurrence.
Defining a certain amount of core parameters (5-7) for every error might be very helpful for
technicians. To define core parameters for every error it is important to know what are the external
factors that might influence the occurrence of the error. Also certain product modifications would be
required. Certain parameters, such as limited information on media, is already available in data
dumps but is not yet displayed, so it only requires including this information into visualization.
Parameters such as temperature and humidity are not available yet, and to add them it would
require certain modifications of the products, such as adding more sensors.
Temperature and humidity in the area where product is located might also play a significant role in
defining the causes for the problem. Every FST when asked whether addition of such parameters
would help insisted that it would be a highly appreciated and valuable addition. As mentioned by one
29
of the FSTs, he had a customer who had 2 of the same Océ products on different floors in the
building, ground floor and the upper floor. The product on the ground floor has constantly had
problems with paper jams, whereas product on the upper floor has hardly ever had such a problem.
After few visits with a purpose to solve paper jams issues, FST noticed that humidity in the room
where problematic product was placed was causing paper in the printer to become wavy which
caused paper jams. But it was difficult to convince customer this was the case as customer believed
the problem is in the product itself. In such cases having additional sensors integrated in the products
which would register those additional parameters would be helpful. Those parameters might also be
displayed in a graphical way over time, which would give an impression about the state of
environment on a certain point in time. Due to the lack of certain business cases I would advise to
study the added value of this proposal before implementing it.
Algorithm improvement
Algorithm used to put errors on worklist was criticized by some FSTs as in some cases it puts on
worklist errors that actually are not the root causes. And by fixing them the issue that customer was
complaining about might not disappear at all or disappear temporarily. Improving algorithm might be
another proposal for the future, but to improve that algorithm it has to become more product
specific. Every product has its own errors and own factors that influence occurrence of those errors.
Current algorithm is generic, what allows it’s usage on all machines without adjustment, so it is a
question whether making it product specific is a better solution.
Optimization for touch screen
The prototype developed is not optimized for the usage on touch screen panels that is an important
disadvantage as Océ is planning to use future data visualization on touch screen panels mounted into
products. The proposals described in current thesis do not exclude usage of touch screen panel; it is
only the prototype that has been developed without consideration of that technology.
30
Appendix 1
Current STAR version
Service Tooling for Analysis and Reporting (STAR)
STAR is a software package that is used by Océ to maintain equipment released to the field.
STAR tooling consists of four main elements:

Analysis – allows to view details of every error;

Archive Explorer – archive of STAR. Stores data on all products over time;

Logbook Explorer – stores reports provided by FSTs after their maintenance visits;

SMART. Star Maintenance Advisory and Reporting Tool – produces the Work List (see
Picture 3 “Worklist”).
The focus of current assignment was on Analysis, Logbook Explorer and SMART tools, which allows
FST to view information on errors over time.
SMART
SMART is an advisory tool for FSTs which generates worklist. This tool performs a comparative
analysis between data from three different sources:

Logbook – keeps all information on previous FST visits;

Current product status (data dump) – represents data on current values of counters, events,
modifications;

Product knowledge database – database with predefined data about the product, such as
thresholds.
Picture 3 “Worklist”
31
After data from all sources is processed a Worklist is being created. Worklist reflects issues
(errors, warnings) that are highly recommended for FST to pay special attention at. Worklist is not a
guide for actions, but more a set of advices that are intended to assist FST on his maintenance
activities. With an Analysis tool user can get a better overview of the product performance as well as
occurrence of errors.
Analysis
Analysis tool allows having a more detailed overview of all errors that occurred within time span of
last seven FST visits (7 snapshots). With analysis tool following data can be viewed:

General information (visit date, previous visit dates, software version etc.)

Counters values;

Error and warning overviews;

Error history and trends.
Currently Analysis tool represents data in a way that disallows FST to get a quick and informative
view over it. Data is shown in tables which imply certain amount of thinking before any conclusion
might be drawn.
Current structure of Analysis tool requires FST to invest additional time in order to detect patterns
and get an idea of when, how often does a certain error occur and what could be the possible
reasons.
Errors and warnings
There are multiple types of alerts that user has to pay attention at. The most common alert is
an error. Errors occur in a case of a flaw of the system, which does not allow the product to perform
its tasks as intended. There are two types of errors: permanent errors and errors. Errors appear in
case of a malfunction and require some action from the user in order to bring product back to full
functioning. Permanent errors block the usage of a product, and can be resolved only by Field Service
FST. Permanent errors always appear on the top of the error list as most critical.
Another type of alert is warning. Most common reason for occurrence of warnings is that
value of counter is approaching its maximum, and by means of warning it signals that a particular
part will soon have to be changed.
Algorithms for pattern recognition
Current software uses multiple data dimensions to assist FST on performing maintenance
activities. Displaying data on the errors that have already happened is a way to fix something that is
already broken. It would result in a much more efficient and cost-effective performance if it is
possible to forecast and prevent threats before they actually happen.
For this purpose Océ has implemented algorithms, which take into account occurrence of a
certain error over time, total usage of the function and relative occurrence of an error over time to
32
determine whether there is a trend in error occurrence or not, and if yes, then whether it is negative
or positive.
Relative occurrence of an error is counted by dividing total usage with absolute occurrence of an
error. For example if error occurred 20 times, while 100 pages have been printed, relative occurrence
would be 5, 1 error per 5 pages.
There are two basic analyses used to evaluate performance of the product. The tail analysis
allows comparing on an error code level the performance of one product with performance of the
rest of population of the same class products. The trend analysis compares current relative
occurrence value to the same values but in past periods. If the value of current relative occurrence is
lower than the past values than there is a negative trend, which means that product performs worse
than before. Results of both analyses are shown to the user in Analysis and SMART tools of STAR
software.
Appendix 2
Interview 1
Report
Interviews of Océ FSTs within context of an internship assignment
1-5-2013, Konstantins Babahodzajevs
Service Processes & Applications
Introduction
Interviews have been conducted with a purpose to define disadvantages and highlight benefits of the
current approach used for visualization of data extracted from Océ products as seen by Field Service
FSTs, as well as find and shape yet unutilized opportunities.
Taking in consideration limitations in time and availability of FSTs as well as amount of interviewees
needed for conclusions to be valid, it has been decided to interview 2 to 3 FSTs from each of DP and
WFPS and of different experience level.
For comments on every separate interview, summary of concerns and proposals of FSTs interviewed
you can go to the section Summaries of interviews.
33
Description of approach
Why these FSTs?
It has been decided to go for 3 representatives of both departments, DP & WFPS and request
interviews with best FSTs in opinion of their managers. Within the list of the best FSTs, we’ve chosen
equal amount of experienced (over 10 years) and less experienced (less than 10 years) (Simon A. H.,
1991).
Qualitative, semi-structured interviews were conducted.
Semi-structured interviews imply that there is no set of particular questions but only an interview
guideline, which outlines topics to be covered, and interviewee has a freedom on how to reply.
One of the goals was to let them understand, that we are not changing much, data stays the same, but it is
shown in a more effective and efficient way.
Purpose is to try to get a “natural” input from FST about their concerns on STAR tooling +TSM, without giving
them a certain direction. My role is just to keep them within the scope, and try to follow the guideline.
Important points are listed below:

Introduction of myself and assignment (Tech part not administrative)

Describe planning of all 3 meetings, ask about holidays in July.

“Facesheet”:

Name, age, gender, position in company, number of years employed, number of years as FST, number
of years as DP FST…).

What are the overall impressions about the STAR?
a.
What would you like to have more?
b.
What is obsolete and/or unnecessary?
c.
What data from data dump is not yet used?
d.
Is there any data missing in Worklist? Analysis?
e.
How could the view inside an error be represented? What factors included?
f.
What time span and area scale should be set by default?
g.
Specifics of maintenance of your products? (WFPS, DP, CP)
h.
Visual/graphical or textual/dropdown?
Limitations/Concerns:
How reliable are the answers and opinions of FST?
They are used to the current interface as it is present for the last 15+ years. People are not willing to
implement and use new different interfaces (based on scientific paper). After prolonged usage of an old
interface, they might not see new hidden opportunities.
34
Summary
After analyzing results of interviews I came up with a list of most common concerns of FSTs which
should be tackled and result in proposals for improvements for new visualization strategy (list of
concerns below is not prioritized or sorted in any way):
-
Unified Dashboard. Amount of clicks to be done, separate applications to be launched, and
launch time was a most common concern. FSTs have to launch too many separate
applications in order to perform maintenance. It requires time and effort to launch every
next application, which is unproductive, costly and might be frustrating.
-
Graphical representation - current STAR tooling lacks graphical representation. Most of
tables currently used within STAR tooling should be kept, as they are helpful and FSTs are
used to them, but certain information, such as error overview over time and population
should be viewed via graphs with a purpose to give a better and clearer overview.
-
Sorting, filtering, search capabilities are very poorly introduced within STAR tooling. With
contemporary search algorithms available, these functions can be easily implemented, what
could significantly ease and fasten work of FST.
-
Show on demand unused information – some information (full list of consumables,
unimportant (resulting) errors) should be either removed or hidden with ability to show on
demand, as it is not used by FSTs and can act as a potential reason for confusion and waste
of time.
-
Ability to reset counters in SMART – some parts are changed on counter value, in this case
counter has to be resetted and currently it can be only done through TSM, which requires
extra time and effort, due to the fact it is a separate application with a user unfriendly
interface. Adding ability to reset an appropriate counter within SMART (dashboard) would
reduce maintenance time and avoid frustration.
-
Running tests from STAR (dashboard) – currently STAR provides error overview and analysis,
but to run tests that show whether an error is true or not FST has to launch TSM. This
inconvenience can draw FST away from using STAR, as TSM can provide error overview and
analysis as well, but with limited capabilities + tests. Option allowing running tests directly
from tooling should be introduced.
-
Error prioritization/removal – most of errors that pop-up on a worklist are errors that are
consequences of other, root errors. Those errors should not appear on a worklist, as with
fixing root error those errors will disappear. Error prioritization is required in order to show
most important (process) errors with a growing trend on top, with least important errors and
incidents not appearing at all, or being hidden with an ability to be shown on demand.
-
Overview and comparison between errors over population – FSTs find it helpful to see what
has been done about the error by other FSTs on different products. Another dimension is to
see occurrence of an error over population of same products for an analysis purpose.
-
Product settings (T71) are not always accessible, but plays very important role. At this point
no specific solution has been found for this issue, except of an advice to fix the bug.
35
As mentioned in section “Description of approach”, there are certain limitations that should be
considered when evaluating and analyzing results of interviews. All FSTs have different levels of
experience comparing them to each other, but they all are rather experienced employees overall.
This results in a fact they have developed certain habits and methodologies on performing their
maintenance activities that limit their abilities on being objective when evaluating current tooling
and being asked to provide their ideas on what could be added and improved within current tooling.
Rejection of new technologies and not willingness to implement changes is a common problem
within employees of all kinds of companies.
Taking in consideration this remark, FSTs still managed to provide important feedback and shape the
requirements for new STAR tooling version. First proposals are described in a next section.
Proposals
This section will provide short description of ideas and proposals that are aimed on solving concerns
mentioned by FSTs and spotted while exploring STAR tooling. This section is intended to provide an
overview of ideas without going in details.
Dashboard
Introduction of dashboard is an inevitable step that has to be taken. Océ Remote Service is already a
step in that direction. It implies interface that uses tabs, but my proposal would be to use blocks. As
an example a dashboard called “My Cockpit” used by Procter&Gamble can be used.
36
Every block would contain certain information, such as consumables, error list, updates (overall
updates: modifications, news; personal updates: messages from other FSTs). In the middle as a most
insightful tool, Analysis would be placed. Some blocks, which are not vital for every FST, could be
made adjustable, meaning blocks could be hidden or shown on demand, which would allow FST to
avoid information that he does not find helpful. Certain important information would be in fixed
blocks that can’t be hidden. Dashboard would save settings of every FST and restore it with every
new connection of that particular FST.
Analysis
Current analysis tool includes important information which has to be used by FSTs, but due to its user
unfriendly interface it is inconvenient to read this information what results in being properly used
only by 30% of interviewed FSTs.
Introduction of a graph representing occurrence of an error in multiple dimensions might solve this
issue. Showing error occurrence over time within last 7 visits (snapshots), with ability of choosing
time span (day, week, month, since last visit, 2 last visits etc) and scale (only this error over this
product, only area 14 errors in this product, only all errors in this product overall, only area 14 errors
over all vp6* products etc). Previous FST visits would be reflected on the graph as well (green
bubbles on the graph), with an option to see logbook from that visit by clicking on the bubble.
Hovering over a certain bar on the graph would trigger a pop-up with more detailed information on
what errors and how many times occurred on that day.
37
The graph represented above would be multidimensional, scalable and easy to read, what is
supposed to allow FSTs to get a faster and more informative idea on error occurrence and nature.
The graph would as well play a consolidation role, allowing to show error date, occurrence over time,
logbook etc. what excludes necessity in opening multiple windows and additional effort.
Information can be compared by overlaying graphs on one another (e.g. try to see relation between
occurrence of error in 02 and 05 by overlaying graphs and see whether they occur at the same point
in time).
TSM link might be stored within every red square accordingly to an appropriate error.
Consumables
Depending on the type of consumable (change part on value or not) it is shown or hidden upon a
demand (implementation of a multi-layered approach). It can be decided to show all consumables
that are attached to the parts which are changed on value, and make a colored backgrounds for
every consumable, showing how close to max value it is (green – <75%, yellow 75-99%, red 100+%).
With this approach it can be estimated when the part will have to be changed as current value, max
value and average usage per day are known. Current approach might allow FST to plan in advance
next visit to the customer, or even replace part in advance (when it’s over 85% for instance) if cost of
extra visit is higher than exchanging part in advance (for every part percentage has to be set
separately, depending on its cost and cost of service visit).
With a purpose to avoid FSTs changing parts in advance, which might result in financial losses for the
company, but making them to change parts only when it is economically and time wise acceptable
for company as well as convenient for the customer, a short report could be introduced. When part
is changed before it has reached its threshold, a pop-up appears asking to provide a reason for
changing the part in advance. Through this it will be possible to monitor which parts have been
changed, when and for what reason.
Filtering, sorting, search. Counter reset. Hiding information. Error prioritization
Concerns mentioned above are technical and can be solved through implementing additional
functionality to the tooling through enhancing the code.
For error prioritization it is necessary first to define process errors as ones appearing on top. Another
condition for appearing is whether error trend is growing, if not, error should not appear.
For resetting counter, after FST has entered a part that he has changed in the SMART, a button Reset
Counter has to be enabled next to the field where part has been entered.
Product settings
A link to product settings might be displayed in a Product Data block. After clicking on
aforementioned link window will pop up, which will show current settings and upon request all
changes that have been done in the recent past (if it is possible it might as well show who has applied
changes).
38
Summaries of interviews
FST 1
Works with hardware issues;
Uses: Archive Explorer, Logbook, Remote Dispatch, SMART, STAR Analysis, TSM, Robin, Outlook
Remarks: Logbook -> History -> Errors/Actions – error description is missing.
Proposals: overview of population of products for back office analysis; present values of parameters
relevant to an error, priority for errors, graphical overview of occurrence.
FST 2
Works with hardware issues;
Uses: Archive Explorer, Logbook, Remote Dispatch, SMART, STAR Analysis, TSM, Robin, Outlook
Workflow:
SMART
Archive Eplorer (Logbook,
Backup) + TSM
Remarks: TSM shortcut never used, no additional information required, too much information shown
(does not need all counter values); Error history (histogram) almost never used; worklist is often
incorrect
Proposals: show only counters that require part change on value, ability to see what exactly has been
done by a previous FST, but not just “done”; prioritize errors, do not show logbooks older than 6
months, ability to have an immediate connection (phone) to other FSTs (especially the previous one)
FST 3
Works with hardware issues;
Uses: Archive Explorer, Logbook, Remote Dispatch, SMART, STAR Analysis, TSM, Robin, Outlook
Workflow:
SMART
Archive Eplorer (Logbook,
Backup) + TSM
39
Remarks: Important to see occurrence of errors over time, to determine the root error (uses Error
Overview -> All Errors In Time); Never uses TSM link, SEA, consumables list, history + top tabs in
Logbook Explorer; no additional information required; occasionally modifications do not pop-up
Proposals: Unified dashboard, ability to reset counters from SMART.
FST 4
Works with hardware issues;
Uses: Archive Explorer, Logbook, Remote Dispatch, SMART, STAR Analysis, TSM, Robin, Outlook
Workflow:
Archive Eplorer
(Logbook, Backup) +
TSM
SMART
Call
STAR Analysis
Remarks: TSM shortcut never used, no additional information required, SEA highly appreciated, too
many clicks to be performed, History + all above tabs in Logbook are not used
Proposals: Unified dashboard, graphical representation (visual representation would give a fast idea
on presence of trend and its nature)
FST 5
Works with software issues
Uses: Archive Explorer, Logbook, Remote Dispatch, SMART, TSM, Robin, Outlook
Workflow:
Call
Archive Eplorer (Logbook,
Backup) + TSM
Remarks: Almost never uses STAR Analysis, no additional information required
Proposals: Unified dashboard
FSTs 6/7
Tech. support specialists;
Use: Archive Explorer, Logbook, Remote Dispatch, SMART, STAR Analysis, TSM, Robin, Outlook
Remarks: want to know what has been done about the same errors on other products (population
overview); T71 often does not work; want to see trend graphically; too many not important error
40
codes (remove unimportant rather than prioritize); improve conditions for errors to appear on
worklist.
Proposals: if no growth in error occurrence -> no service visit; improved search/sort/filter
possibilities within applications; unified dashboard.
FST 8
Works with hardware issues;
Uses: Archive Explorer, Logbook, Remote Dispatch, SMART, TSM, Robin, Outlook
Workflow:
Call
TSM
Archive Eplorer (Logbook,
Backup) + TSM
Remarks: STAR lacks tests, therefore uses LotusNotes instead of STAR Analysis as TSM include error
list and tests at once; uses Logbook for manual overview on occurrence.
Proposals: direct link to tests from SMART, ability to add pictures, e-mails in SMART report, show
only important errors that are not consequences of other errors, sort errors by area, ability to extract
graphs for escalating purposes.
Interviews 2
While second round of interviews FSTs expressed their overall thoughts over proposals I proposed.
All FSTs confirmed the directions taken and provided remarks and advices on proposals could be
improved even further. All remarks have been considered and included in the final version of
prototype.
41
Appendix 5. Definitions of criteria used in evaluation model
To evaluate the visualization a SUS model have been chosen, but after studying the model it has
been decided to expand model with an additional criteria to Learnability and Usability, namely
Memorability.
As defined by Brooke (Brooke J., 1996), usability can be defined as appropriateness to a purpose of
any particular artifact. The same have been confirmed by a usability testing specialist from Océ
Design, who defined usability as suitability to purpose it serves for.
Learnability – can be defined as an ability of visualization to teach user on its behavior. As an
example in the case of current visualization hovering might be used. When user hovers over a bar in
the graph, a table with more detailed info on that day appears. After certain amount of appearances
user will learn this behavior.
Memorability – was described by Océ Design usability testing specialist as a consistency throughout
the system. If once again hovering option is applied, then keeping this behavior throughout the
system, in other words staying consistent, allows user to expect certain actions from the system as a
response to a certain action from him.
42
10 Bibliography
Bouvier D., C. T. (2012). User Interface Evaluation by Novices. ITiCSE.
Brooke J. (1996). SUS - A quick and dirty usability scale.
Brown J. (1996). Methodologies for the creation of interactive software. Victoria University of
Wellington.
Bryman A., B. E. (2011). Business Research Methods. Oxford University Press.
Chen M., E. D. (2009). Data, Information, and Knowledge in Visualization. Visualization Viewpoints ,
12-19.
Davis F. D. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information
Technology. University of Michigan.
Dos Santos B. L., B. L. (1988). A Study of User Interface Aids for Model-Oriented Decision Support
Systems. Management Science.
Engers T. M. (2001). Knowledge Management: the Role of Mental Models in Business Systems Design.
Hahn J., S. R. (2000). A framework of knowledge management systems: issues and challenges for
theory and practice. Carlson School of Management University of Minnesota.
Iliinsky N., S. J. (2011). Designing Data Visualizations. O'Reilly.
Jeffries R., M. R. (1991). User Interface Evaluation in the Real World: A Comparison of four
techniques. ACM Computer Human Interaction.
John E.B., K. E. (1996). Using GOMS for User Interface Design: Which technique? ACM Transactions on
Computer-Human Interaction.
Karat C., C. R. (1992). Comparison of empirical testing and walkthrough methods in user interface
evaluation. Association for Computing Machinery.
Lewis C., P. P. (1990). Testing a Walkthrough Methodology for Theory Based Design Walk-Up-and-Use
Interfaces. Association for Computing Machinery.
Lewis R.J., S. J. (2009). The Factor Structure of the System Usability Scale. Human Computer
Interactions.
Nielsen J. (1994). Usability Inspection Methods. Association for Computing Machinery.
Pintelon L.M., G. L. (1992). Maintenance management decision making. Elsevier Science Publishers
B.V.
Radner R. (2000). Costly and Bounded Rationality in Individual and Team Decision-Making. Industrial
and Corporate Change , 623-658.
43
Rowley E. D., R. G. (1992). The Cognitive Jogthrough: A Fast-Paced User Interface Evaluation
Procedure. Association for Computing Machinery.
Simon A. H. (1991). Bounded Rationality and Organizational Learning. JSTOR .
Simon A. H. (1979). Rational Decision Making in Business Organizations. The American Economic
Association.
Smith L.S., M. N. (1986). Guidelines for designing user interface software. Userlab Inc.
Tsang A. H.C. (1995). Condition-based maintenance: tools and decision making. The Hong Kong
Polytechnic University.
44