Developing new model for
website assessment
Applying evaluation theory
approach for assessment of
Slovenian government
websites
Boris Kragelj
University of Ljubljana
Faculty of social sciences
[email protected]
The “idea” of evaluation study approach
Evaluation is...
... systematic use of procedures, methods and research
techniques for assessment of ideas, action plans and results of
“public” programs (Rossi and Freeman 1993, 5)
...that provides answers to the following questions:
- How well is program (of government websites) conducted?
- Does program (of government websites) functions in the way it supposed
to?
- How well program (of government websites) achieves desired goals?
- Why desired goals (of gov. websites) are not achieved and how it is
possible to achieve them?
- What kind of changes did program (of gov. websites) reach and what are
its effects?
We applied this idea of the evaluation to the program of
goals of Slovenian government websites.
Construction of evaluation study
A three step model:
Phase one:
PROGRAM
- Clear identification of websites’ goals, mission and strategies for
achieving them.
- Determination of program of Slovenian government websites
Phase two:
ANALYSIS
- Complete analysis of websites and measurement of its
performance in all aspects related to its goals and strategy:
accessibility, usability, utility…
- Holistic analysis of Slovenian government web sites of all
relevant aspects of Slovenia government activities on the WWW
Phase three:
ASSESSMENT
- Comparson of the results of analysis with the websites’ program
through the seven criteria of evaluation theory: effectiveness,
efficiency, adequacy, equity, responsiveness, appropriateness and
economy.
- Comparison of the condition of Slovenian government websites
with their program and strategic goals. Making judgments about
how well the program achieved desired goals.
Phase one: PROGRAM
On the highest level we determined the program of Slovenian government websites
in the following way:
-
-
Presentation of the government bodies, offices and providing transparency of
their activities (information service, contacts, government documents,... Online)
Providing government services online (tax declaration, car registration, change
of address...)
Providing possibilities for citizen’ participation in the process of government
activities (through the e-mail, forums and web polls...)
Making communication and interaction with government bodies and offices
simple and fficient
Generally accessible government for all citizens with no social and technological
discrimination
High care for security and privacy when using information and services on
government websites
Phase two: ANALYSIS
•
Matrix of different aspects of websites, and methods, research techniques that
were used for analysis of Slovenian government websites
Guideline
Web Usability Expert HTML
Log
review, Mystery Performance
survey testing review validation analysis Feature shopping
test
inspection
Content
x
Usability
x
x
Traffic
x
x
Accessibility
x
Performance
x
Responsibility
x
Customer
x
satisfaction
Security&
x
privacy
•
GENERAL RULE: The more aspects included in analysis and the more different
methods applied for the analysis – the better assessment in the final stage of
evaluation study is possible!
Phase two: ANALYSIS
Combining methods is an important advantage
COMBINIG
METHODS
QUALITY OF
ASSESSMENT
INDIVIDUAL
METHOD
DIFFERENT METHODS APPLIED,
DIFFERENT ASPECTS ANALYSED
Phase three: ASSESSMENT
•
General evaluation criteria for the assessment of programs and their
application on the program of Slovenian government websites
Criteria
Description
Application to SLO government websites
Effectiveness
Was desired effect achieved?
Was desired effect of SLO government web
sites achieved?
Efficiency
How much effort was needed for
effect achieved?
How much effort was needed for the effect of
SLO government websites achieved?
Adequacy
To which extent was (with achieved
effect) problem solved?
To which extent was (with achieved effect)
problem of SLO government websites solved
Equity
Are expenses and benefits equally
distributed among different groups?
Are expenses and benefits for SLO government
websites equally distributed among different
groups?
Responsiveness
Are political effects of program in
accordance with needs, preferences
and values of parties involved?
Are political effects of SLO government
websites in accordance with needs, preferences
and values of parties involved?
Appropriateness
Are desired effects really important?
Are desired effects of SLO government websites
really important?
Economy
Were resources spent in the right and
appropriate way?
Were resources for SLO government websites
spent in the right and appropriate way?
RESULTS
of evaluation study
Major results:
•
•
•
Complete and relevant assessment of websites on the basis of their mission,
achievement of certain strategic goals
Identification of major problems (with the websites) for realization of the
program of websites
Directions and orientation for future developments of websites in the way of
reaching the strategic goals
Additional value:
•
•
•
•
•
Point of departure for revision of the program and strategy of websites
Ground for measuring the progress toward the realization of the website
program and its success in reaching the strategic goals
Practical guidelines for improvement of websites
Establishing best practices
Providing ideas for reasonable changes in further developments
RESULTS
of evaluation study in the case of Slovenian government websites
•
•
Slovenian government websites are not meeting any of the evaluation criteria to
the greater extent and are not Effective, Efficient, Adequate, Equal, Responsive,
Appropriate, neither Economic
The web sites (with regard to their program) therefore are not achieving their
mission as they are:
–
–
–
–
–
•
•
The major problem presents very poorly defined program of Slovenian
government websites, what finally results in desynchronized functioning of
government websites
As an additional value of evaluation approach some important solutions for
renovation of Slovenian government websites were provided:
–
–
•
not generally accessible to all citizens,
Unfriendly and very difficult to use
lacking of the right content, information and services,
they take very little care for privacy and security matters
and they are without appropriate, common appearance
A document with guidelines for essential reformation and improvement of Slovenian
government websites was prepared
Draft of the strategy for development of Slovenian government on the WWW was
launched
Groundwork for measuring the progress toward the mission
Conclusion
One final question:
Is evaluation study approach for website assessment
applicable to all different kinds of websites, or can it
be used only for assessment of government
websites?
This dilemma will remain unsolved until the next case of assessing
websites with evaluation study approach from another area will
be tested!
Developing new model for
website assessment
Applying evaluation theory
approach for assessment of
Slovenian government
websites
Boris Kragelj
University of Ljubljana,
Faculty of social sciences
[email protected]