INCREASING USER ENGAGEMENT
IN ICT-BASED
SOCIAL INNOVATION PROJECTS
Nikolaos Kampanos
Master Thesis
Master of Science in Business Informatics
Department of Information and Computing Sciences
Faculty of Sciences
Utrecht University
This Master Thesis submitted in partial fulfilment of the requirements
for the degree of Master of Sciences in Business Informatics
Author
Nikolaos Kampanos
4299124
Utrecht University
[email protected]
First Supervisor
Second Supervisor
dr. Sergio España Cubillo
Information and Computing Sciences
Utrecht University
[email protected]
dr. R.L. Slinger Jansen
Information and Computing Sciences
Utrecht University
[email protected]
Department of Information and Computing Sciences
Faculty of Science
Utrecht University
1
I would like to dedicate this thesis to my grandmother Paraskevi Tselentis
who gave me the opportunity to pursue my Master's degree, as I wanted to.
2
3
Executive Summary
Social Innovation is a domain of innovation that has become very popular over the last
decade. All over the world, social challenges like the growth of population, climate change,
migration, unemployment and diversity need an innovative social solution. Lack of User
Engagement has been identified by the authors as one of the problems that social
innovation projects face.
The author of this thesis did a systematic literature review in order to explore the current
knowledge on the severity of the user engagement problem as well as on the ways to
measure and increase user engagement on those projects. On top of that, a survey-based
research has been designed and conducted in order to see how the situation in projects in
the real world is and which of the ways from the literature are used and what the
successfulness of them.
The findings of the research conducted proved to be of particular interest. From the social
innovation projects that participated in the research more than 71% stated that they face
from a moderate to a serious user engagement problem. The literature review has yielded
12 and 24 distinct ways to measure and increase user engagement, respectively. But
according to the results of the survey, such ways are far from widely known and applied;
around half of the ways were not used by 40% of the projects or more. Ways to measure
user engagement using online behavior metrics and self-reported engagement were more
popular than ways measuring the cognitive engagement. Gamification elements as ways to
increase user engagement, which are popular in the scientific literature, were on average
the least popular ways used to increase user engagement.
A repository containing these methods has been created. An automated tool has been
designed and implemented to assist the social innovation project owners to deepen their
knowledge of the ways to increase user engagement and to assist them to select the bestfitted ways to increase user engagement on their projects.
4
5
Table of Contents
Executive Summary ............................................................................................................... 4
List of Tables ......................................................................................................................... 8
List of Figures ........................................................................................................................ 9
Chapter 1
Introduction ..................................................................................................... 11
1.1.
Social Innovation ................................................................................................. 11
1.2.
User Engagement................................................................................................. 13
Chapter 2
Research Methodology .................................................................................... 15
Chapter 3
Conceptual Framework .................................................................................... 18
Chapter 4
Literature Research Protocol ............................................................................ 21
4.1.
Research Questions Specification ......................................................................... 22
4.2.
Source Selection .................................................................................................. 22
4.3.
Primary Studies Identification .............................................................................. 22
4.4.
Primary Studies Selection..................................................................................... 24
Chapter 5
Results of the Literature Review ....................................................................... 26
5.1.
Citizen Science Projects ........................................................................................ 26
5.2.
Behavioral Change Projects .................................................................................. 27
5.3.
User Engagement Model and its attributes .......................................................... 28
5.4.
Extracted ways to Measure and Increase User Engagement ................................. 30
5.4.1.
Measure User Engagement .......................................................................... 31
5.4.2.
Increasing User Engagement ........................................................................ 34
Chapter 6
Survey .............................................................................................................. 39
6.1.
Survey Object ...................................................................................................... 39
6.2.
Survey Planning ................................................................................................... 40
6.3.
Survey Designing.................................................................................................. 40
6.4.
Selecting Participants ........................................................................................... 40
6.4.1.
The “Old School” Way .................................................................................. 40
6.4.2.
The inbound marketing way ......................................................................... 41
6.5.
Chapter 7
Data Analysis ....................................................................................................... 41
Implementation of the user engagement method repository ........................... 50
7.1.
Technology and design......................................................................................... 50
7.2.
Online Repository ................................................................................................ 53
Chapter 8
Treatment Validation ....................................................................................... 54
Chapter 9
Discussion ........................................................................................................ 56
9.1.
Main Conclusions ................................................................................................. 56
6
9.2.
Research Limitations ............................................................................................ 58
9.3.
Future Work ........................................................................................................ 60
Chapter 10
References ................................................................................................... 62
Chapter 11
Appendix...................................................................................................... 67
11.1.
Ways to Increase User Engagement: Characteristics ......................................... 67
11.2.
Ways to Measure User Engagement: Characteristics ........................................ 67
11.3.
Survey .............................................................................................................. 68
11.4.
Email template ................................................................................................. 74
11.5.
Reminder email ................................................................................................ 75
11.6.
Reminder email for those that have clicked the survey link .............................. 75
11.7.
Treatment Validation Survey ............................................................................ 76
7
List of Tables
Table 1: Volunteer User Profiles .......................................................................................... 26
Table 2: Definitions of the attributes of engagement........................................................... 29
Table 3: Ways to Increase/Measure User Engagement Characteristics ................................ 31
Table 4: Comparison Mailchimp to Personal Email .............................................................. 41
Table 5: Social Innovation Project Categorization ................................................................ 45
Table 6: Online Repository: Use Level Record ...................................................................... 53
8
List of Figures
Figure 1: Research Method visualization using PDD ............................................................. 17
Figure 2: Archimate Basic Concepts ..................................................................................... 18
Figure 3: Generic Conceptual Framework ............................................................................ 19
Figure 4: Social Innovation ICT Projects Conceptual Framework .......................................... 20
Figure 5: Systematic Literature Review visualization using PDD ........................................... 21
Figure 6: Primary Studies Identification Visualization using PDD .......................................... 23
Figure 7: Knowledge Domains ............................................................................................. 24
Figure 8: Primary Studies Selection visualization using PDD ................................................. 25
Figure 9: Fog Behaviour Model ............................................................................................ 27
Figure 10: Activation and Continuation Phase in Behavioral Change Projects....................... 28
Figure 11: User Engagement Model..................................................................................... 29
Figure 12: Conduct a Survey Method visualization ............................................................... 39
Figure 13: Project's Participants .......................................................................................... 42
Figure 14: Project's Location................................................................................................ 42
Figure 15: Project's Maturity Level and Maturity Level Definitions....................................... 42
Figure 16: Problem Severity ................................................................................................ 43
Figure 17: Ways to Measure User Engagement ................................................................... 43
Figure 18: Ways to Increase User Engagement .................................................................... 44
Figure 19: Project Categorization......................................................................................... 45
Figure 20: Crowdfunding: How successful are these ways to Increase User Engagement ..... 46
Figure 21: Community Tool: How successful are these ways to Increase User Engagement . 47
Figure 22: Civic Engagement: How successful are these ways to Increase User Engagement 47
Figure 23: Social Mapping: How successful are these ways to Increase User Engagement ... 48
Figure 24: Sharing Economy: How successful are these ways to Increase User Engagement 48
Figure 25: Automated System Model .................................................................................. 50
Figure 26: Automated System - Beta Version ....................................................................... 51
Figure 27: Automated System: Sorting tool ......................................................................... 52
Figure 28: Treatment Validation Questionnaire ................................................................... 54
9
10
Chapter 1
Introduction
The main research objective of this MSc. Thesis is to provide a comprehensive report on the
problems related to user engagement in social innovation ICT projects and additionally,
propose an automated system to assist the project owners to select the best-fitted methods
or tools for their need. Chapter 1 elaborates on the domains of Social Innovation and User
Engagement while in Chapters 2 and 3 the research methodology and the conceptual
framework are presented. Chapters 4, 5 and 6 explain the research protocol and the results
deducted from the literature review and the survey, respectively. The implementation of the
proposed solution and its validation can be found in Chapters 8 and 9. Finally in Chapter 10
the discussion of this research is exhibited, entailing the main conclusions, limitations and
stimuli for further scientific research.
1.1.
Social Innovation
Over the last years, Social Innovation has started to gain ground as a way to face social
challenges. The first use of the term “Social Innovation” was used in scientific work in the
1960s, while there are also reports using the term on the 1970s. However, Social Innovation
started to be studied seriously in the 1990s. There are many definitions available right now
in the literature (Bunchball Inc., 2010; Mulgan, 2012; Phills, Deiglmeier, & Miller, 2008;
Tushman, 1977). A combination of the following two definitions is going to be used as the
definition of Social Innovation in this research.
“Social Innovations are innovations that are social both in their ends and in their means. In
other words: it covers new ideas (products, services, and models) that simultaneously meet
socially recognized social needs (more effectively than alternatives) and create new social
relationships or collaborations, that are both good for society and enhance society’s
capacity to act” (Mulgan, 2012).
According to Phills et al. (Phills et al., 2008) social innovation must not be related to the
private value, but they should contribute to the distribution of financial and social value
toward society as a whole. Therefore, according to these authors, social innovation is “a
novel solution to a social problem that is more effective, efficient, sustainable, or just than
existing solutions and for which the value created accrues primarily to society as a whole
rather than private individuals”.
The aforementioned definitions have been combined to provide a clearer and more
concrete definition about the Social Innovation projects which the author of the thesis chose
to elaborate on. The new definition should contain the elements of “social both in their ends
and in their means” and the “solutions with the value created accrues primarily to society as
a whole rather than private individuals”. The outcome of that combination is defining Social
Innovation as:
11
“A social Innovation is a novel, more effective solution to a social problem that is
social both in its ends and in its means, with the value created accruing primarily to
society as a whole rather than private individuals”
Potentially, every novel solution that solves a social problem can benefit in the end the
society as a whole, as societies consist of individual citizens. Therefore, by improving
individually the citizens of society, the society benefits, as well. However according to the
previously mentioned definition, this is not the projects and innovations that this research is
focusing on. An example of two projects solving the same social problem of assisting blind
persons perform variant tasks are presented to make the distinction clearer.
There is an innovative ICT project, named BlindSquare1 that helps blind persons identify
objects using image recognition algorithms and GPS. This is a very innovative solution for a
group of individuals – persons with limited vision ability - in society. Until that point
everything is inlined with the definition. Moreover, if all the blinds are using it then it would
be easier for that group of individuals to interact with the part of the society, adding a social
layer to the app. However, this solution is not social in its ends and neither the created value
accrues primarily to the society. Those two characteristics are missing for the specific project
to be among the ones that this research is focused on.
On the other hand, there is another ICT project called “bemyeyes”2 that connects blind
persons with non-blinds ones through video chat helping them identify objects or assisting
them with different tasks. This is a case of social innovation ICT project, social both in its
ends and means. The outcome could be perceived individually for the blind persons
completing their tasks, although the value for the society is more than that, creating
communities of volunteerism, having interactions between the two parts of the society,
increasing the awareness of the life and the capabilities of blind people
Social Innovation can be applied to many different societal challenges. (Mulgan, Tucker, Ali,
& Sanders, 2007) have proposed a list of them:
-
-
-
1
2
Climate Change: demands new thinking on how to reorder cities, transport systems,
energy and housing to dramatically reduce CO2 emissions. Technology has a decisive
role to play—but so will social innovations that help to change the behavior of
people.
Growing diversity of countries and cities: demands innovative ways of organizing
schooling, language training and housing to prevent segregation and conflict.
Stark inequalities: have widened in many societies, such as the UK, the US and
China, and tend to be associated with various other social problems, ranging from
violence to mental illness.
Rising incidence of long-term conditions: such as arthritis, depression, diabetes,
cancer and heart disease (which are now chronic as well as acute), demanding novel
social solutions as well as new models of medical support.
http://blindsquare.com/
http://www.bemyeyes.org/
12
-
-
1.2.
Rising life expectancy: which requires new ways of organizing pensions, care and
mutual support, new models of housing and urban design (for 4- and 5–generation
families and continually changing housing needs), and new methods for countering
isolation.
Behavioral problems and affluence: including obesity, bad diet and inactivity, as
well as addictions to alcohol, drugs and gambling.
Difficult transitions to adulthood: requiring new ways for teenagers to successfully
navigate their way into more stable careers, relationships and lifestyles.
Happiness: the mismatch between growing GDP, stagnant well-being, and declining
real welfare according to some measures requires new ways of thinking about public
policy and civic action.
User Engagement
Social Innovation as it was mentioned before has to be social in both of its ends. That makes
most of the projects depending on the communities of users that are going to be created. A
social community needs to have a minimum number of participants in order to foster a
sense of belonging among the participants. Some ICT projects are depending their survival
on the number of users that use the project on a regular basis (Lin, 2014).
Most of the times the users are not limited to passive participation and active involvement is
required on their part creating a sense of ownership of the project. In (Batey, Bull, &
Decorme, 2013; Bruns, 2014) those users have been mentioned as “produsers” or
“prosumers” as users that also produce content for the projects or consumers that also
produce. There is also the case that there is a distinction between the ones who contribute
to a project and the ones who benefit from that (e.g. the beemyeyes app or a project to
assist refugees). In (Tarnawska & Ćwiklicki, 2012), those users are clustered to “Direct
Consumers” and “Indirect Consumers”, and the ones that benefit are called “Beneficiaries”.
In this research, the term “Beneficiaries” is also going to be used to describe the ones that
benefit from a social project and the term “Contributors” for the ones that contribute to the
project.
The 1-9-90 rule from (Nielsen, 2006) states that in most communities 90% of the users only
benefit from the community and never contribute, 9% contribute a little and only the 1%
assist in a decent way. Based on that knowledge, for a project to survive and be viable in the
long term, project owners have to find ways and methods to increase user engagement in
their projects.
“User Engagement is a quality of user experience that depends on the aesthetic appeal,
novelty, and usability of the system, the ability of the user to attend and become
involved in the experience. Engagement depends on the depth of participation the user
is able to achieve with respect to each experiential attribute.” (O’Brien & Toms, 2008).
According to (Butler, 2001), to participate in an online community, as those described above,
the perceived benefits of using the system should be more than the costs of participation.
Although most of the times, in social innovation projects there is no monetary reward or any
other types of tangible rewards for the contributors, the perceived benefits are most of the
times the feeling of doing something good, the belief to a cause, the sense of belonging to a
13
community, etc. There are different theories describing the factors that can motivate a
person to participate in a social innovation project. The results from the literature review
that has been conducted concerning those theories are presented in the following list.
-
Trust Theory(Rdings 2002) (Lin, 2014)
Social Cognitive Theory (Kamal, Fels, McGrenere, & Nance, 2013; Lin, 2014)
Commitment Theory (Lin, 2014)
Social Network Theory (Lin, 2014)
Social Influence Theory (Lin, 2014)
Information System Success (Liaqat & Bhatti, 2015)
Uses and Gratification Theory (Kamal et al., 2013)
Human Computer Interaction (Tomayess & Pedro, 2014)
Fluency Theory (Coursaris & Van Osch, 2016)
Cognitive-Affective Model of Perceived User Satisfaction CAMPUS (Coursaris &
van Osch, 2016)
The energy consumption domain is one of most researched domains concerning user
engagement and psychology of participation in ICT project targeting to decrease the
consumption. Despite that research work, many related projects still face problems with
engaging users (Goldbach & Gölz, 2015), which leaves room for much work to be done
towards that direction. As social innovation is gaining more attention and as the theories of
increasing user engagement start being researched more and more, then probably this
decade may be the decade of engagement (MacDonald, 2013).
14
Chapter 2
Research Methodology
Based on the above research objective the following research questions were formulated
following the approach of (Wieringa, 2014), who recommends writing research questions as
questions RQ1, RQ2, RQ4 and the technical research questions as actions RQ3.

RQ1: How severe is the user engagement problem in Social Innovation ICT Projects?
o RQ1.1: Are they any differences in the nature of the problems?


RQ2: What existing methods are suitable to face the user engagement problem?

RQ3: Create a repository containing the best practices or tools that have been
collected and an automated system to assist the problem stakeholders to select the
best-fitted practices or tools to solve their problem.
RQ4: To what extent does the automated system serve its purposes of helping
problem stakeholders select the best-fitted methods or tools for user engagement?
To investigate the previously mentioned research questions, a research method has been
created based on the design science cycle as defined by (Wieringa, 2014). The full cycle
consists of five steps.
1. Problem investigation: Defines which phenomena must be improved and why. Based
on this, goals are specified for the artifact to accomplish.
2. Treatment design: In this phase, the requirements are specified and it is investigated
if the requirements contribute to the goals specified earlier. Furthermore, it is
investigated which available treatments there are already and based on the
gathered data, the artifact is designed.
3. Treatment validation: Here the artifact is validated for satisfying the requirements.
Furthermore it is tested if any other effects, if any are observed in the validation,
apply in the specific context of the validation. The simplest way to validate an
artifact is by expert opinion.
4. Treatment implementation: It is defined here as the application of the treatment to
the original problem context.
5. Implementation evaluation: The goal of the implementation evaluation is to
evaluate the implemented treatment in the original problem context.
This project did not involve all five steps mentioned above. From these steps, only the first
three was part of this master thesis project. The last step will be continued in new follow-up
projects. In more details the following tasks have been implemented.
 Investigation of current problems on user engagement (RQ1).
A systematic literature review has been conducted to collect all the required data to build a
solid scientific background for the research. Further analysis of the systematic literature
review can be found in Chapter 4.
15
In addition to that, a survey and some oral interviews were be used, acquiring quantitative
but also qualitative data concerning current problems on user engagement. Those surveys
had as a basis the results of the previously mentioned literature review. An in-depth analysis
of the survey can be found in Chapter 6. (Problem Investigation)
 Exploration of existing solutions and available methods to increase and measure user
engagement (RQ2).
Part of the previously mentioned systematic literature review was to identify existing
solutions and methods to increase and measure user engagement. Based on those results,
some additional questions was used in the survey that was mentioned before, targeting to
collect best practices currently used in the real world. In section 6 an extensive description
of the method and the construction of the survey can be found. (Treatment Design –
Available Treatments)
 Development of the automated system (RQ3).
All the collected ways and tools have been stored in a repository. The author developed an
automated system which assists the problem stakeholders to select the best-fitted ways for
their problem. (Treatment Design – Treatment Design)
 Performing a validation (RQ4).
A validation of the benefits and the drawbacks of the automated tool has been implemented
using a questionnaire. (Treatment Validation)
A graphical representation of the aforementioned research methodology has been created
using the Process-Deliverable Diagram (PDD) method (Van de Weerd & Brinkkemper, 2008)
and it is presented in Figure 1.
“A process-deliverable diagram (PDD) consists of two integrated diagrams. The left-hand
side of the diagram is based on a UML activity diagram, and the right-hand side of the
diagram is based on a UML class diagram. ” (Van De Weerd, Brinkkemper, Souer, &
Versendaal, 2006)
The three steps of the design science cycle have been depicted using the activity notation.
Each one of them has different sub-activities, which produce the concepts that appear on
the right part of the diagram. The relationships between the concepts are presented using
connectors.
16
Figure 1: Research Method visualization using PDD
17
Chapter 3
Conceptual Framework
In this chapter, the conceptual framework of this research is presented. This conceptual
framework has been created based on the knowledge and experiences of the author on
social innovation projects3 and it has been reviewed and validated by a group of experts,
was consisting of some Greek social entrepreneurs, the directors of the Impact Hub 4 in
Athens and in Hague and the people In charge of social innovation projects in two
foundations for entrepreneurship in Greece.
In order to create the conceptual framework of the current thesis, the basic principles of
ArchiMate have been used. ArchiMate is an open and independent enterprise
architecture modeling language to support the description, analysis and visualization of
architecture within and across business domains in an unambiguous way.
Motivational concepts are used to model the motivations, or reasons, that underlie the
design or change of some enterprise architecture. It is essential to understand the factors
influencing the motivational elements, which are often referred to as drivers. An assessment
is defined as the outcome of the analysis of a driver. Assessments are associated with
stakeholders, with the latter being either some individual human being or a group of human
beings, such as a project team, enterprise, or society. The actual motivations are
represented by goals, principles, requirements and constraints. Each goal represents a
desired result. Principles and requirements represent desired statements of need that must
be realized by a system. Constraints represent restrictions on the way in which a system is
realized (Haren, 2012). The aforementioned concepts are presented in the following figure.
Figure 2: Archimate Basic Concepts
In Figure 3, the generic conceptual framework of a Social Innovation Project is presented.
The figure begins with the stakeholders, that could be either one or more, who have their
motivations – drivers – to change a current situation affecting the society. Therefore, their
goal is to solve a social problem. To do so, they have to create a Social Innovation Project
(S.I. project). This S.I. Project may or may not include a Social Innovation ICT Project (S.I. ICT
project) aiming to support the project. In this thesis, only the S.I. Projects that include S.I. ICT
projects are going to be researched. From now and on the S.I. ICT projects are going to be
referred to as Projects.
The specific concept of S.I. ICT projects – projects - is going to be the main part of interest
for this research. A more detailed conceptual framework of it is presented in Figure 4.
The author has participated in the one year honors program “Young Innovators League” offered
from the university of Utrecht, where he worked in a social innovation project.
4
http://www.impacthub.net/
3
18
Figure 3: Generic Conceptual Framework
In Figure 4 the conceptual framework of an S.I. ICT project is presented. The S.I. Project
Owner and Team are now stakeholders of the S.I. ICT project and their common driver is to
have a successful project. In order to have a successful project, a good product by itself is
not enough. A user growth strategy is also needed.
"Just because you build a great product does not mean you'll just get users."
By Dharmesh Shah
So the initial driver of having a successful project is determined by both a good product and
an efficient user-growth strategy.
A drive for a good product is transformed to a goal of creating good IT infrastructure. This
goal is divided to two separate sub-goals of building good IT infrastructure and effectively
testing it. Those goals are going to be implemented by actors, being the S.I. ICT project team
– from now and on Project Team – and the testing users.
An effective growth strategy is divided into two sub-drives referring to the two aspects of
user growth- user acquisition and user retention.
When examining the aspect of user acquisition, a potential and a possible constraint is low
acquisition rates. To overcome this problem a marketing team is assigned with the task to
increase acquisition rates, by implementing different marketing strategies, namely
commercial campaigns and earned or gained publicity being a few examples.
As far as user retention is concerned, stakeholders aim to retain the users that have already
acquired. In other words, they are trying to keep users’ satisfaction high. In this point the
driver of some other stakeholders – externals ones – is unveiled. So the users of the project
want to be more satisfied, and the same do the S.I. Project Owner and Team. To achieve
that, three assessments and their corresponding goals are described.
“User Engagement leads to Retention, which leads to Growth.”
by Paul M. Boyce
Increasing User Engagement, in particular, is the main target of this thesis and is going to be
researched and analyzed more in further part of the thesis.
19
Figure 4: Social Innovation ICT Projects Conceptual Framework
Increasing the beneficiary and contributor users are part of the Marketing Team’s tasks.
Here, users, who are external stakeholders of the project, have been divided into two
different concepts, Beneficiaries, and Contributors. The difference between those two
concepts is based on the same concept which describes the difference between customer
and consumer.
“By definition, a customer is someone who buys services or goods from someone else while
a consumer is someone that consumes a certain product or commodity” (Julita, 2011).
Therefore, in an S.I. Project there are the Beneficiary users, who are the users that enjoy the
main benefit from the specific project and the Contributor users, who contribute to the
project so the Beneficiaries can indeed benefit. There is a motivation – engagement strategy
to keep Contributors engaged, which is going to be analyzed more in depth later on.
For example, let’s take an S.I. Project aiming to help blind persons to see. In this context, a
S.I. ICT Project has been developed, where a blind person – the Beneficiary User – can hold
their phone in front of his/her eyes and a non-blind user – the Contributor User –
volunteering can describe through the phone what the blind person “sees.”
20
Chapter 4
Literature Research Protocol
Part of the problem investigation and the treatment design of the design science cycle that
have been described in Chapter 2 was to conduct a literature review. The approach
proposed by (Kitchenham, 2004) for systematic literature reviews has been followed. A
systematic review is a means of evaluating and interpreting all the available research that is
relevant to a particular research question, topic area, or phenomenon of interest. It aims to
present a fair evaluation of a research topic by using a trustworthy, rigorous, and auditable
methodology. It involves several stages and activities, which are briefly explained below:



Planning the review: the need for the review is identified, the research questions
are specified and the review protocol is defined.
Conducting the review: the primary studies are selected, the quality assessment
used to include studies is defined, the data extraction and monitoring are performed
and the data obtained is synthesized. An activity to test the reliability of the review
protocol was added during this stage.
Presenting the results: A review report is presented.
The aforementioned activities are presented in the following PDD.
Figure 5: Systematic Literature Review visualization using PDD
21
4.1.
Research Questions Specification
In order to examine the current problems in user engagement on social innovation IT
projects, the existing ways on facing those challenges and criteria to measure the quality of
the proposed solutions, the following research questions were formulated:

RQ1: How severe is the user engagement problem in Social Innovation ICT Projects?
o RQ1.1: Are they any differences in the nature of the problems?

RQ2: What existing methods are suitable to face the user engagement problem?
The intention of these research questions was to understand the size of the user
engagement problem and to collect all the reported problems and solutions concerning user
engagement on social innovation ICT projects. The result of the research was used as the
base for the survey that has been conducted to social innovation ICT project participants to
investigate how the situation in the real world is.
4.2.
Source Selection
Two types of search methods were used to select appropriate and representative papers.
The first type, automatic searching, was based on the scientific paper database Scopus and
Google Scholar.
The second one, manual searching, was applied to verify the correctness of the automatic
review and to carry out a more in-depth study exploring new trends and approaches that
may not have been yet reported in published scientific work. To do that the following
sources were used:



4.3.
Reference lists from the primary studies and the reviewed articles.
Journals, gray literature (i.e. technical reports, work in progress).
The Internet.
Primary Studies Identification
The research questions were used as a basis for the creation of the search strings with which
to identify primary studies. So the outcome was three search strings dividing the literature
review into three knowledge domain sections: i) User Engagement in Social Innovation, ii)
Ways to Increase User Engagement and iii) Ways to measure User Engagement. In order to
finalize the previous search strings, different versions of the initial search strings have been
tested containing various combinations of synonyms or order of the main words (social
innovation, user engagement, measure user engagement, increase user engagement). The
result was to identify more than 1500 pieces of scientific work as primary studies. As it was
mentioned before three search strings were formulated, the full description of these search
strings along with their primary results are the presented in the next paragraphs.
The steps of the method which have been used in order to identify those primary studies are
depicted in the following PDD.
22
Figure 6: Primary Studies Identification Visualization using PDD
Search String A:
 "user engagement" AND ("social innovation" OR "social entrepreneurship" OR
"social enterprises" OR "social projects")
The search string consists of two parts: the first part links those works that describe user
engagement, the second part limits down to user engagement on social innovation projects
and synonyms of those. As a result, 486 primary studies have been identified.
Search String B:
 ("increase user engagement" OR "increasing user engagement" OR "boost user
engagement" OR "user engagement strategies") AND ("social innovation" OR
"social entrepreneurship" OR "social enterprises" OR "social projects")
This search string consists of keywords focusing on increasing user engagement and narrows
down on social innovation projects. However, the outcome was only 37 papers, so a new
string was tested without having the restriction of the social innovation.
23
Another reason for selecting the second version was that the author would like to extract
additional ways of increasing user engagement which haven’t been associated with social
innovation projects yet. When the following search string is finally being used gives an
outcome of 864 primary studies.

("increase user engagement" OR "increasing user engagement" OR "boost user
engagement" OR "user engagement strategies”)
Search String C:
 "measuring User engagement."
Using the same way of thinking as before the search string to explore the ways to measure
user engagement has been constructed. This search string is consisted by keywords focusing
on measuring user engagement. The specific string is going to be used as it is without
merging with the “social innovation” keywords because it refers to general measuring ways.
The outcome of this search was 402 pieces of scientific work.
In the following figure, a representation of the different knowledge domains and their
relations are presented.
User Engagement Knowledge Domain ~ 1.8m results
String A
Social
Innovation
String C
Measuring
Methods
ICT
Projects
String B
Increasing
Methods
Figure 7: Knowledge Domains
4.4.
Primary Studies Selection
Having a pool of more than 1500 pieces of scientific work, made the selection of the ones
that were actually going to be used in the research a mandatory task. The first step in that
direction was to create an initial filter by screening the keywords and the abstract of each
one of them. An additional step was to filter the projects according to the inclusion and
exclusion criteria. The final step was to remove any duplicate results. The selection
procedure is visualized in Figure 8.
The main inclusion criterion is that the projects mentioned in the papers should be ICT
projects. On the other side, books, poster publications, and papers not written in English
have been excluded.
24
Figure 8: Primary Studies Selection visualization using PDD
As a result of the selection process, 102 papers were used in the research. The rate of
acceptance is about 6% to 7%, however as it was mentioned before the main inclusion
criterion is that the projects mentioned in the papers were ICT or simplified IT projects. ICT
or IT could not stand alone in the keywords that which were used for the string selection
because they are acronyms and would probably create "confusion" to the search engine
prompting it to included results with the “it” subject or part of any word containing the
letters “it” or “ict”.
25
Chapter 5
Results of the Literature Review
The first and most important finding of the systematic literature review that has been
conducted is the lack of the literature on the subject of User engagement in Social
Innovation projects. Of course, both scientific domains have started recently enjoying proper
attention from the scientific community. Although there are a lot of familiar elements in
other types of projects which the user engagement element have been researched with
decent results. In the next subsections, some of those kinds of projects are presented.
5.1.
Citizen Science Projects
A citizen science project is a Project, most of the times of ICT nature that recruit ordinary
citizens to execute human computations tasks. Citizen Science in simple words is a
partnership between scientists and common people willing to contribute voluntarily to a
greater scientific cause (Ponciano & Brasileiro, 2014). Thus, there are a lot of similarities with
the social innovations projects, where the individuals in co-operation with other
stakeholders are willing to contribute voluntarily to a project to solve a social problem, with
the value created accruing the greater community primarily.
NASA’s Clickworkers5 platform is an example of an ICT citizen science project, which enables
ordinary people to contribute by marking craters in high-resolutions images of the surface of
Mars (Bruns, 2014). Other similar projects are the Galaxy Zoo and The Milky Way Project,
which have been operating for almost 2 years and have managed to involve more than a
thousand of participants executing more than one billion executed tasks. That gave the
opportunity to (Ponciano & Brasileiro, 2014) to identify two general categories of behavior
and 5 more specific volunteer user profiles based on the contribution of the participants.
In one hand, there are “real volunteers” having in their conscious to spend a certain amount
of time to a cause. That behavior is called Volunteerism Behavior and includes more
dedication and cognitive effort when they participate in the project. On the other hand is
the Helping Activity Behavior where the contribution is made in a more sporadic way. Of
course, the best way of behavior is the first one because most of the times they contribute
the most to large projects (Ponciano & Brasileiro, 2014). In the following table, the
characteristics of the five aforementioned volunteer user profiles are presented.
Hardworking
Spasmodic
Persistent
Lasting
Moderate
Activity
Ratio
High
High
Low
Low
Intermediate
Variation in
periodicity
Low
Moderate
High
High
Intermediate
Activity Duration
Short
Irregular
Long
Long
Intermediate
Participation
Time
Short
Short
Long
Short
Intermediate
Table 1: Volunteer User Profiles
5
http://www.nasaclickworkers.com/
26
As it has been presented there are different kinds of volunteer behavior from users within
the content of a project. That means that if the project owners want to keep high
engagement levels and make the most of each specific categories they have to adjust and
personalize their engagement ways. More about the engagement ways is going to be
presented in subsection 5.4.2.
5.2.
Behavioral Change Projects
One category of projects that the scientific community has researched a lot is the Behavioral
Change Projects. Those projects are targeting to change the behavior of the user (den
Ouden, 2012). The ones that we are going to present are dealing with environmental
challenges (e.g. energy consumption, smart energy, energy saving) (Batey et al., 2013; Castri,
De Luca, Lobsiger-Kägi, Moser, & Carabias, 2014; Geelen, Reinders, & Keyson, 2013;
Goldbach & Gölz, 2015; PieterValkering, Erik Laes, Kris Kessels, 2014) or are related to with
the health (chronic diseases, better lifestyle) (Baghaei et al., 2009; Miller, Cafazzo, & Seto,
2014; Paulo & Barreto, n.d.). As it was mentioned before, not all of those projects can be
called social innovation projects no matter if they are dealing with environmental challenges
or health challenges. Although the research on increasing user engagement that has been
made on those projects, it can also be applied to social innovation projects.
Fog (Fogg, 2009) proposed a framework for behavioral change based on 3 key elements:
Motivation, Ability, Triggers. Therefore, in order to perform a target behavior, there are
certain prerequisites; the user should be sufficiently motivated, have the ability to perform
the behavior and should be triggered to perform the behavior. These three factors must
occur at the same moment. Otherwise, the behavior may not happen. As it can be seen in
Figure 9, each of the aforementioned prerequisites has certain sub-categories.
Figure 9: Fog Behaviour Model; adopted from [Fog, 2009]
27
More specifically, the proper behavior trigger should be used for a certain behavior to be
caused, depending on the core motivator and the simplicity factor. For examples, if the
motivation is high, but the simplicity factor is low then a facilitator is the most common
trigger. A facilitator could be a tutorial video, an interactive guide or a discount that could
increase the simplicity factor. On the other hand, if the simplicity factor is high but the
motivation is low then a spark should be used for an example a notification from the
application. Lastly, if both the motivation and the simplicity factors are high, then a simple
signal should be enough to make the user perform the activity, such as a regularly settled
email notification.
For example, there is an application where the citizens of a town report problems about
their city. So the target behavior is that users report a problem through an application form.
To increase the likeliness of performing the target behavior, there should be high motivation
and the application should be simple enough. If the report is simple to be done and the
motivation is high, then a simple trigger can be used to motivate the user to make the
report. For example, in the case that the form is too complicated, then a facilitator can be
used, giving instructions on how to do it.
5.3.
User Engagement Model and its attributes
Fog presented a behavior model identifying the factors that affect the user to perform a
target behavior. However the goal of every project is the user to stay engaged with the
project and ideally, the participant could stay involved with the project as a new habit for
him/her. In (PieterValkering, Erik Laes, Kris Kessels, 2014) they presented a graph depicting
the previous goal. They separated the user participation into two phases: the one is the
activation phase where the user started to participate to the app and the second is the
continuation phase where the user remains engaged and still participates in the project. To
that point a transformation on his/her behavior might have happened.
Figure 10: Activation and Continuation Phase in Behavioral Change Projects; adapted from [Valkering, 2014]
In order to understand the attributes that keep the user engaged through the activation and
continuation phase (Brien & Toms, 2010; Lalmas, O’Brien, & Yom-Tov, 2014; O’Brien &
Toms, 2008) created and tested a user engagement model. The user engagement process
can be analyzed in 4 stages: Point of engagement, the actual engagement, the point of
disengagement and re-engagement.
28
Figure 11: User Engagement Model; adapted from [O’Brien, 2006]
The point of engagement is where all starts; it is the point when the user is attracted to the
project, which could be a result of the content or the design of the project. From that point
and on, the system has to retain the user attention level high,so the user attains the specific
goals of the project. This phase is called sustained engagement. Different kinds of attributes
affect this procedure, attributes like aesthetics, affect, attention, challenge, control,
feedback, interest, motivation, novelty, perceived time (Banhawi, Mohamad Ali, & Mohd
Judi, 2012; Brien & Toms, 2010). In Table 2 the definitions of those attributes are presented.
When the sustained engagement period ends, then disengagement follows. If the system
had succeeded to create high user engagement, then the changes of reengagement are
increased. If this cycle keeps having a stable period, then there are high chances of passing
from the activation phase to the continuation and to the ultimate goal of changing the
behavior of the user.
Attributes
Aesthetics
Affect
Attention
Challenge
Control
Feedback
Interest
Motivation
Novelty
Perceived Time
Definitions
Visual beauty or the study of natural and pleasing (or aesthetic) computer-based
environments (Jennings, 2000).
“The emotional investment a user makes in order to be immersed in an
environment and sustain their involvement in the environment” (Jennings, 2000).
The concentration of mental activity; concentrating on one stimulus only and
ignoring all others.
The amount of effort experienced by the participant in performing an online task.
How “in charge” users feel about their experience with the technology.
“Information that is sent back to the user about what action has been done or
what result has been accomplished” (Stone, Jarrett, Woodroffe, & Shailey, 2005).
The “feeling that accompanies or causes special attention to an object or class of
objects”
Elements that bring about focus or a desire to proceed with an activity (Jennings,
2000)
Variety of sudden and unexpected changes (visual or auditory) that cause
excitement and joy or alarm (Aboulafia & Bannon, 2004)
Users’ perception of estimated time spent on task.
Table 2: Definitions of the attributes of engagement
29
5.4.
Extracted ways to Measure and Increase User Engagement
One of the objectives of the conducted literature review was to be able to answer the second
research question:
 RQ2: What existing methods are suitable to face the user engagement problem?
Although the research on ways to increase user engagement seems enough to answer the
second research question, this is not the truth. It is obvious from the scientific side, at least,
that something cannot be increased if the state before and after the implementation of the
treatment is unknown. So this research answered this question by focusing in two directions,
the ways to measure and the ways to increase user engagement in a social innovation ICT
project.
As a result, 12 ways to measure user engagement and 24 ways to increase user engagement
were revealed from the literature. A list containing all these ways can be found in the next
paragraph and a further analysis of each way can be found in sections 5.4.1 and 5.4.2.
Ways to Measure User Engagement
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
Interviews
Think aloud
Questionnaires
Mouse Tracking
Eye Tracking
Psychophysiological Measures
Absence Time
Click Through Rate
Click Depth
Return Rate
Conversion Rate
Average Reading Time / Dwell Time
Ways to Increase User Engagement
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
Engage boundary spanners of the targeted communities
Involve users in creating the online communities
Engage users on the media platform they are already using
Provide analytics to the users
Maximize control on the ICT project for the user
Provide Monetary Rewards / Discounts
Give responsibilities to the users
Set Goals and Achievements
Use Reminders
Give Feedback
Award Badges
Award Points
Use Rankings and Leaderboards
Give In-App Awards and Prizes
Use Challenges and Contests
Use Levels
30
17.
18.
19.
20.
21.
22.
23.
24.
Use virtual goods
Social Comparison
Use the Social Networks
Use Up-votes
Use Likes
Increase users’ reputation by their contribution
Communication among end-users
Use a combination of low and high tech platforms
The aforementioned ways have been researched using the following criteria:
-
Type [Method, Technique, Guideline]
Supporting Tool [ None, Module/Plugin, Standalone Software]
Level of Validation [ Not at all, Ad-hoc, Case study, Experiment]
As it can be seen in Table 3, most of the ways to measure user engagement were
techniques, did not have any supporting tool and were validated using case studies. On the
other hand, the ways to increase user engagement were guidelines, with no supporting tool
and with no level of validation. A per way analysis of the characteristics can be found in
appendices 12.1, 12.2 .
Inclusion Criteria
Type
Supporting Tool
Level of Validation
Method
Technique
Guideline
None
Module/Plugin
Standalone Software
Not at all
Ad-hoc
Case study
Experiment
Ways to Measure
User Engagement
20%
67%
13%
100%
0%
0%
20%
0%
73%
7%
Ways to Increase
User Engagement
24%
0%
76%
98%
2%
0%
35%
31%
24%
10%
Table 3: Ways to Increase/Measure User Engagement Characteristics
5.4.1. Measure User Engagement
As it was mentioned before, a user engagement model able to measure user engagement
before and after the implementation of them, is crucial in order to understand the success
of the ways to increase user engagement. In that direction three researchers - Munia
Lalmas, Heather O’Brien and Elad Yom-Tov - have done a lot of work over the last years with
many publications (Brien & Toms, 2010; Dupret & Lalmas, 2013; Lalmas et al., 2014;
Lehmann, Lalmas, Yom-Tov, & Dupret, 2012; O’Brien & Toms, 2008). The approaches to
measure user engagement can be divided into three main groups: self-reported
engagement, cognitive engagement, and online behavior metrics (Lehmann et al., 2012).
5.4.1.1. Self-Reported Engagement
This type of method is commonly used in social sciences where individuals express their
attitude, feelings, and beliefs about a subject (Lalmas et al., 2014). The advantages of using
this method are:
31
-
Flexibility in appliance a variety of settings
Convenience in administration
Ability to proceed with a statistical analysis of the received data
Ability to examine larger populations
Some ways that are used for that are:
Interviews
Can be used in a structured or unstructured way, interviewing one person or more as a
group.
Think aloud
Is a way to capture the experience that user has a while or right after he had interacted with
the system we want to measure. It is a way to have a closer view of how they feel with
specific parts of the system.
Questionnaires
It can be used either as quantitative measuring instruments or as qualitative, every option
has a trade-off between quantity and quality. (Jacques, 1996) has proposed a 13-items
questionnaire, (Webster & Ho, 1997) has altered it by adding 2 items and (Brien & Toms,
2010) have proposed the User Engagement Scale, a questionnaire including 31-items.
However the results depend on user subjectivity and that is something that could cause bias.
5.4.1.2. Cognitive Engagement
This specific category measures engagement by observing and collecting data of the users
during their participation in the project. The cognitive processes that take part during the
participation can be measured using a variety of new ways that are analyzed further on in
this section. The benefits of measuring user engagement with these ways are:
-
The received data are more objective
Can be performed on the time the task is processed without interrupting the user.
Can provide more in-depth information about how the user feels while he interacts
with the system
Some ways that are used are:
Mouse Tracking
Capturing the average number of clicks/slides/drags/mouse-overs on hotspots that reveal
hidden (or embedded) information upon users’ interaction (Dupret & Lalmas, 2013; Oh,
Bellur, & Sundar, 2015).
Eye Tracking
The process of measuring either the point of gaze (where one is looking) or the motion of an
eye relative to the head (Lalmas et al., 2014; Oh et al., 2015).
Psychophysiological Measures (Heart Rate, Temperature Sensors, Electromyographic)
This type of method is based on the state that every reaction of our body corresponds to a
psychological process (e.g. when we get embarrassed, our cheeks get red and warm) (Lalmas
et al., 2014).
32
Measuring cognitive engagement, although objective, is suitable for measuring only a small
number of interaction episodes at close quarters. Also, the equipment some time could be
expensive and until now comfort level at the time of measuring, is not as high as it is
supposed to be.
5.4.1.3. Online Behavior Metrics
The last category utilizes the collected metrics that are related to the online behavior of the
users. Different kinds of measures like how much time the user stayed in the system or how
many times the user used the system in order to compare with other data from other users
and create some patterns. Patterns that can be used to create user profiles and personalize
the system to their needs or to improve parts of the systems that may be problematic
according to user’s data (e.g. high dropout ratio in a specific page or task). The advantages
of using this method are:
-
Easy to implement and interpret
Ability to compare many things with the same data
Ability to collect data on a larger scale with low resources
Some ways that are used are:
Absence Time
The absence time is simply the time elapsed between two sessions of a user. According to
(Dupret & Lalmas, 2013) if users find a site interesting, engaging or useful, they will return to
it sooner.
Click Through Rate
Number of clicks (e.g., on an ad) divided by the number of times it was shown (Lalmas et al.,
2014; Oh et al., 2015).
Click Depth
Average number of contiguous pages viewed within a site (Lalmas et al., 2014).
Return Rate
Numbers of returns to the platform within a session (Lalmas et al., 2014).
Conversion Rate
(Mostly for e-commerce): fraction of sessions which end in the desired user action (e.g.,
purchase) (Lalmas et al., 2014).
Average Reading Time, Dwell Time
The average time that a user spent on a website per session or per user (Lalmas et al., 2014;
Oh et al., 2015).
Of course there is not right or wrong, depending on the size of the population that is going
to be measured, the available resources in time, experience and money, a different method
can be used. Measuring user engagement is something new and is still being in progress,
although measuring before and after the implementation of any ways targeting to increase
user engagement is mandatory.
33
5.4.2. Increasing User Engagement
If this decade is the engagement decade [11], then sure the answer to the question “How to
increase User Engagement” is a hot topic. From the literature review, 24 ways to increase
user engagement have been revealed. All these ways are presented in this section,
categorized in 6 groups.
5.4.2.1. Online Communities
Trying to create online communities with a certain level of contribution and participation is
challenging (Chang, Kaasinen, & Kaipainen, 2012; Nielsen, 2006; Niragira & Adeyeye, 2013)
trying although to build online communities from already existing offline communities could
be even more challenging because of the smaller amount of possible users (Chang et al.,
2012). The following solutions have been proposed to the specific user engagement problem
from different researchers:
Engage boundary spanners of the targeted communities
Boundary Spanners bridge the structural hole between two clusters conceptualized as being
separated by a boundary of some sort (Tushman, 1977). In simpler words, a group of
persons that can share and explain your innovation to a targeted community.
Involve users in creating the online communities
For example giving the ability to users to invite other users to the online community, create
subgroups, naming the groups etc. (Lin, 2014).
Engage users on the media platform they are already using
Using a platform that already is used by the community that you are targeting, instead of
trying to educate and convince the users to use a new one (Vila, 2014). For example a
Facebook page, or messenger to report problems to the municipality.
5.4.2.2. Motivational factors
Another category is the motivational factor, referring to the factors that motivate the users
in a behavioral change project but could also be applied in social innovation projects.
Provide analytics to the users
The users are motivated when they receive insights about themselves or the cause they are
called to participate (Goldbach & Gölz, 2015). Analytics can increase user engagement and
loyalty more than anything else that you do. Users at all levels love to have information
about them. Provide analytics to the users, referring to their usage of the app, giving the
opportunity to discover things about themselves (Emerson, 2011).
Maximize control on the ICT project for the user
Give to the user the ability to control the platform, by having a profile page, control their
privacy settings and customize parts of the interface (Goldbach & Gölz, 2015).
Provide monetary rewards/discounts
Any monetary reward can increase the motivation for the users (Farzan et al., 2008;
Goldbach & Gölz, 2015). Although especially in social innovation projects, monetary awards
are rarely given, an alternative to that could be virtual economic rewards or earning
discounts as it was proposed from (Lee, Marks, Kern, & Jordan-cooley, 2013).
34
Give responsibilities to the users
Giving responsibilities to the users can be strong motivational factors. Users start getting
initiatives and they are trying to do their best (Goldbach & Gölz, 2015). In the case of a
forum or a wiki, for example, it could be to appoint some users as moderators or
administrators. In a community, they can approve new members.??
Set goals and achievements
The users are motivated by goals, especially if they are clear, short-term goals that actually
can be achieved (den Ouden, 2012; Farzan et al., 2008; Goldbach & Gölz, 2015; Kwon,
Halavais, & Havener, 2015; Lin, 2014).
Use Reminders
Use reminders like emails, push notifications and SMS in order to call the user to action
(Goldbach & Gölz, 2015). Messages are effective in encouraging users to stay engaged with
the project (Goldbach & Gölz, 2015; López, Farzan, & Brusilovsky, 2012). Personalized
messages requesting users to perform a particular kind of contribution that follows their
current pattern of participation in the system can increase users’ engagement (López et al.,
2012).
Give Feedback
Informing the users about the cause and the progress of the project so far (Goldbach & Gölz,
2015; PieterValkering, Erik Laes, Kris Kessels, 2014). Most people will voice their concerns
when they feel comfortable and see real benefits (Lam, Chan, Whittle, & Binner, 2011).
Highlight importance of user contribution for the community (Farzan et al., 2008).
Sometimes the feedback should not stop to the usage of ICT methods like emails or in-app
notifications; usage of written ?? information, such mail, leaflets or posters had assist to
increase engagement (Goldbach & Gölz, 2015; Vila, 2014) there were also cases where a face
to face approach has been used.
5.4.2.3. Gamification
The Gamification term was coined in 2002, but it did not gain popularity until 2010. Since
then, it has been with is the hottest word in marketing, education, work and any other
project in need of increased user engagement. Gamification can improve the user
experience and significantly increase both user engagement and user growth. So user
engagement “Is the concept of applying the concept of gaming metaphors in non-game
contexts to influence behavior, improve motivation and enhance user engagement”
(Marczewski, 2013). Including the following strategies not only attract new users rapidly but
also ensures that users do not abandon the project in the long term (Lin, 2014). The concept
of Gamification includes the following game elements.
Award badges
Badges are ornaments or symbols users can earn for pre-defined favorable behaviors
(Emmanuel, Scharf, & Szabadi, 2014; Hamari, 2013; Kumaresan, 2013; Kwon et al., 2015;
Michal, 2015; Miller et al., 2014).
Award points
Reward the users with points for performing various tasks (Emmanuel et al., 2014; Michal,
2015; Miller et al., 2014; Rampoldi-Hnilo & Snyder, 2013).
35
Use rankings and leaderboards
Represent the cumulative results of points collected, challenges won and badges earned
(Emmanuel et al., 2014; Kwon et al., 2015; Michal, 2015; Miller et al., 2014; Rampoldi-Hnilo
& Snyder, 2013).
Give in-app awards and prizes
Give to users awards and prizes that have to do with the gamification elements in the
platform, e.g. unlock new levels, unlock customization for the interface, free elements etc.
(Michal, 2015; Micheletti, 2016).
Use challenges and contests
Challenges are tasks that need to be fulfilled in the time given in order to receive a reward –
a badge or a predefined amount of points (Michal, 2015; Miller et al., 2014; Rampoldi-Hnilo
& Snyder, 2013).
Use Levels
Levels are often defined as point thresholds, so that users can automatically level up based
on their participation, or use levels to indicate status and control access to content on the
site. For example levels are different classes in frequent-flyer programs, colored belts in
martial arts, job titles in industry: an indication that you’ve reached a milestone, a level of
accomplishment in a community and should be afforded a certain amount of respect and
status (Bunchball Inc., 2010; Rampoldi-Hnilo & Snyder, 2013).
Use virtual goods
Virtual goods are non-physical, intangible objects that are purchased for use in online
communities or online games. Users purchase virtual goods like clothing, weapons or
decorations to create an identity for their virtual self while comparing and “showing off”
with their friends (Bunchball Inc., 2010; Lee et al., 2013; Rampoldi-Hnilo & Snyder, 2013).
An extended analysis of the aforementioned game elements can be found in (Michal, 2015).
According to (Rampoldi-Hnilo & Snyder, 2013) there are five principles that will make
gamification mechanics more successful in an ICT project.
-
Selection of one motivational factor
Inclusion of analytics
Simplifying the project
Incorporation of collaboration and social elements
Leverage of mobile device capabilities as part of the gamification strategy.
Some researchers seem to be skeptical about the gamification concept. They believe that
gamification instead of contributing to the ICT project that could distract the users from the
goal of the project. They believe that the users will passionate with the game side of the
project and not the actual cause and will try to cheat the system, in order to succeed in their
goals and missions. Also, there is a chance that when the user completes all the missions or
reaches all the available levels could quit the project (Cramer, Ahmet, Rost, & Holmquist,
2011; Fitz-Walter, Tjondronegoro, & Wyeth, 2011; Michal, 2015).
36
5.4.2.4. Social Layer
The concept of adding a social layer to the ICT projects is also another trend that seems to
work well for user engagement (Baghaei et al., 2009; Chang et al., 2012; Lin, 2014; Miller et
al., 2014; PieterValkering, Erik Laes, Kris Kessels, 2014). A person will be more motivated to
perform a target behavior if he or she can use a system to observe others performing the
behavior (Oinas-Kukkonen & Harjumaa, 2008).
Social Comparison
Comparing user behavior with their friends (Baghaei et al., 2009; Oinas-Kukkonen &
Harjumaa, 2008; PieterValkering, Erik Laes, Kris Kessels, 2014).
Use of the Social Networks
The usage of social networks can be described as a social contagion, due to our
understanding of social norms being variable on the behavior of friends and others that
influence our choices, often on a subconscious level (Batey et al., 2013; Christakis, 2008;
Miller et al., 2014).
Use up-votes
They are generated by the quality of what the user creates, shares or expresses, and they
are awarded by other users (Lee et al., 2013).
Use likes
They are generated by the quality of what the user creates, shares or expresses, and they
are awarded by other users (Lee et al., 2013).
Communication among end-users
Development of ways that stimulate communication among end users. It could be, for
instance, equipped with a discussion or messaging functionality (Geelen et al., 2013).
Increase reputation by their contribution
At a basic level many social networking sites employ reputation-based incentives by
displaying the number of connections and friends a user has (Farzan et al., 2008).
5.4.2.5. Non-ICT Strategies towards User Engagement
Except for the methods, techniques, and solutions that can be applied to the ICT project in
order to increase user engagement, (Vila, 2014) propose some non-ICT strategies that can be
followed from the project’s stakeholders to keep the engagement high between the project
and the cause and the users. Indicatively, such strategies included:
-
Laying a foundation for user-centered design;
Embedding a community organizer into the community of target users;
Inclusion of key stakeholders from the beginning;
Organizing regular events for target users – even if just a small handful will show up;
Linking active participants into a network of volunteers;
When conducting digital outreach, leverage existing relationships.
37
5.4.2.6. Overcoming Technology Barriers
Closing the section of the ways to increase user engagement it has to be mentioned that
especially in social innovation where the problem itself or the solution may include persons
with low relation with IT, the project stakeholders should combine low and high-tech
platforms to promote engagement among the users.
Use a combination of low and high tech platforms
A nice example from Maputo in Mozambique, where the project stakeholders used a wall in
a train station to draw the interface of the project containing the content that they wanted
to deliver. They have left empty space and chalk so the users could interact without having
their own devices or without know how to use one (Vila, 2014).
The systematic literature review that has been conducted to reply to the second research
question, revealed 12 and 24 ways to measure and increase user engagement respectively.
As a first observation, there is a big variation of ways to increase and measure user
engagement available in the literature. A second observation is that, at least on the papers
that have been analyzed in this research, most of these ways were validated in the best case
with a case study. That observation enforces our primary willing to create a survey in order
to investigate which of these ways are known or used in the real world from real social
innovation ICT projects and how successful are they. In the next chapter the design of this
survey is further analyzed.
38
Chapter 6
Survey
As it was described in the research method, a survey has been conducted in order to
examine which of the previously described user engagement ways are used in the real world
from social innovation projects and how successful are they. In the next figure, the method
of conducting a survey that was used in this research, based in (Pfleeger & Kitchenham,
2001), is presented using the PDD technique.
Figure 12: Conduct a Survey Method visualization
6.1.
Survey Object
The basic objects that are going to be searched are the demographics of the social projects,
the existence and the severity of the user engagement problem and the ways used to
increase and measure user engagement.
39
6.2.
Survey Planning
The duration of the survey was one month. The population that was researched consisted
of every social innovation ICT progress that could be found, no matter if they have stopped
their activity or not. The projects that are not working anymore can provide some insights of
the reasons that forced them to stop working, in the best scenario for this research’s
hypothesis would be the low user engagement.
6.3.
Survey Designing
The survey is divided into three main parts, surveying demographics, maturity level and last
but not least, user engagement problem and ways to measure and increase it. As soon as
the first version was ready, it was sent to experts from the social innovation domain,
persons from the academia and some social innovation project owners to been reviewed. All
the remarks have been taken into consideration and as a result the final version was ready.
The final version of the survey can be found in the appendix 12.3 and online at this link.6 The
survey has been designed using the form tool offered for free by Google.
6.4.
Selecting Participants
In order to find and invite social innovation projects to participate in the survey, a twodimensional approach has been followed, with the dimensions further analyzed in the
following subsections.
6.4.1. The “Old School” Way
The “old school” way (Lieberman, 2012), entails two phases, the first one is about the ways
to find all these projects and the second one is about the selling techniques that were used
to convince them to participate.
The “First Phase”
An online research through the "Google" search engine has been conducted using the terms
“social innovation”, “social enterprise”, “social project” and other relative terms. The results
that occurred were investigated to retrieve projects that matched with the definition of
social innovation projects. The projects have also been retrieved from the website of
organizations that incubate that kind of projects or from websites of social innovation
competitions, from the sections of past participants. Personal communication with the
organizers of those competitions and the managers of the incubators has also been made in
order to network more projects. An online directory has been created containing all those
projects with their contact information. The first phase gave a result of 220 projects.
The “Second Phase”
During the second phase two ways were used to reach the selected projects from the first
phase. The first way was to send mass emails to 50 projects using the service of Mailchimp 7
and the second was to send the same email to another 68 projects through my university
email one by one. That was a way to check which of the two ways was the most efficient.
The results pointed out with big difference that the second way is much more efficient.
6
7
www.goo.gl/forms/2LjX4Cj5eC2rZcOQ2
http://mailchimp.com/
40
In Table 4 a comparison between the two ways is demonstrated. The comparison is made
using the data from Mailchimp control panel and a third party add-on for Gmail named
Bananatag8.
Population
Open Rate
Click Rate
(click the link to the survey)
Conversion Rate
(fill out the survey)
Mailchimp (mass emailing)
50
40%
10%
2%
Personal Email (one by one)
55
64%
31%
9%
Table 4: Comparison Mailchimp to Personal Email
The way of sending personal emails one by one was selected for the rest of the projects.
Although the conversion rate was almost five times greater than using Mailchimp was not
enough. To increase the conversion rate, reminder emails were sent two days after the
initial email had been sent. Using the insights from the click rate data a different email was
sent to the project that had previously clicked once on the link to the survey but hadn’t filled
out the survey. All three emails can be found in the appendices. The most effective tool
though, was the telephone calls. After the initial and the reminder emails had been sent,
followed up phone calls were employed, having as result to increase the number of filled out
surveys by 70%.
6.4.2. The inbound marketing way
Inbound marketing ways (Lieberman, 2012), were used in order to attract and convince
social innovation projects to participate in this research and fill out the survey. A website has
been created as well as a Facebook page but with no significant results. The numbers of
visitors and followers were low and none of the answers of the survey came from these two
channels.
In addition to those pages, tweets from the author’s personal account have been made
targeting social innovation events and relative persons. Moreover Linkedin posts on social
innovation groups were posted and the Linkedin InMail option was used to send the survey
to social innovation project founders. Two answers to the survey were collected with that
way.
The lesson learned by trying to use inbound marketing is that just using the existing tools are
not enough effective. Fresh and targeted content has to be posted and a lot of time and
effort has to be dedicated for decent results to be achieved. Having a timeframe of one
month is not enough especially if it is one-man job.
6.5.
Data Analysis
The sample was 220 social innovation projects from all over the world. A number of 44
replies were collected, however only the 39 of them were used in this research. 5 replies
have been excluded because either the survey was not filled out properly or because the
project did not comply with the definition of social innovation as it was defined in chapter 1.
Projects from 15 different countries participated in the research and the majority of them
could be characterized as micro-companies, having less than 10 employees - or more
8
https://www.bananatag.com/
41
precisely- less than 10 project participants. The term project participants is used because
sometimes there is no legal entity behind the project, therefore the parts involved are not
legally employed also no actual employees.
Figure 13: Project's Size
Figure 14: Project's Location
Diversity was found not only on the location base of each project but also on the maturity
level of them. In Figure 15 the description of each maturity level, as it is defined by (Murray,
Caulier-Grice, & Mulgan, 2010), and the number of projects that belong on that level are
presented. The majority of the projects had already an implemented and sustainable
innovation and they were looking for scaling or trying new things on the way they already
are working. That give an extra validation point to our research, meaning that the projects
that participated had already a clear vision about what is working with their projects.
Figure 15: Project's Maturity Level and Maturity Level Definitions
In response to the first research question concerning the severity of the user engagement
problem, 71% of the projects stated that they face user engagement problems from a
moderate to a serious extent. Only 4 projects stated that they do not suffer at all from lack
of user engagement and 7 projects stated that for them it is a minor problem. Below are
presented some statements from the participants about the user engagement problem.
-
“Our main problem is LOW PARTICIPATION or retention.”
42
-
“Number of users determines amount of content (circle process), amount of content
is important to engage users.”
“We always want a higher retention.”
“"Low participation, building a community takes time. No picture uploads or reviews
after experience."
Figure 16: Problem Severity
The literature review has yielded 12 and 24 distinct ways to measure and increase user
engagement, respectively. But according to the results of the survey, such ways are far from
widely known and applied; around half of the ways were not used by 40% of the projects or
more. Ways to measure user engagement using online behavior metrics and self-reported
engagement were more popular than ways measuring the cognitive engagement.
Gamification ways, which are popular in the scientific literature, were on average the least
popular ways used to increase user engagement. The motivational factors and the social
layer were the categories with the most popular and successful ways. More details of each
way can be found in Figure 17 and Figure 18.
Figure 17: Ways to Measure User Engagement
43
In those diagrams the participants of the survey had to state if they knew the way to
increase or measure user engagement and if they used it how successful it is (Not at all –
Very). So on the left side of the Figure, the ways to increase or measure user engagement
are displayed and on the right side there are bars with maximum value the number of the
projects that participated in the survey (in this research this number is 39). So in these bars
the number of participants that do not know the way are displayed with grey color, the ones
that know the way but they not use it are displayed with blue, and the rest are the ones that
use the way and they differentiated with the colors – red, orange, light green, dark green –
according with how successful they found the way.
Figure 18: Ways to Increase User Engagement
One other outcome of this survey is a categorization of the social innovation projects as a
result of the observation that many of the projects that participate in this research are
having some common characteristics. A similar categorization has been made by
(Kacprowicz & Borowiecka, 2014) targeting only e-volunteering projects. E-volunteering and
Social Innovation as domains are presenting a lot of similarities as both have heavy
interaction with users that contribute to the project in a volunteer base. In this research the
categories that have been defined by Kacprowicz was used along with some new ones
defined by this research. In Table 5 the aforementioned categories with their definitions are
presented. In Figure 19 the project’s distribution among these categories is depicted.
44
Figure 19: Project Categorization
Categories
Crowdfunding
Community Tool
Civic Engagement
Social Mapping
Sharing Economy
Do good Things
Crowdsourcing
Microvolunteer
Call for Action
E-petitions
Knowledge Gathering
Marketplace
Time Bank
Description
The practice of funding a project or venture by raising many small
amounts of money from a large number of people, typically via the
Internet (“Crowdfunding,” n.d.). From the crowdfunding category only the
projects that serve a social purpose are included in this research.
A community-based communication tool to strengthen offline & online
communities.
Individual and collective actions designed to identify and address issues of
public concern. It can include efforts to directly address an issue, work
with others in a community to solve a problem or interact with the
institutions of representative democracy (“Civic Engagement,” n.d.).
Social Mapping means creating maps showing some issue. For example:
Local community – when you point places worth visit,
Local government – when show places that need renovation,
Excluded groups – when you create a map of places accessible and nonaccessible for the disabled (Kacprowicz & Borowiecka, 2014).
Sharing platforms, particularly non-profits that are operating to provide a
public benefit. Like sharing one’s home or car, going into strangers’ homes
to do work, or eating food prepared by unknown cooks (Schor, 2014).
Applications that trigger the users to do good deeds.
Is an exchange of knowledge and experiences, process of creating
something new as well as a problem solving distributed to a large,
scattered group of people (Kacprowicz & Borowiecka, 2014).
Simple e-volunteering tasks that be done in short period of time – from 15
minutes to 3 hours (Kacprowicz & Borowiecka, 2014).
Is a call to a database of users to perform an offline social action. For
example the cleaning of a beach or participation to a blood donation
Is an online version of petitions. Again this research focus only on petitions
with a social purpose.
The knowledge gathering type is about creating sources of common
knowledge, free encyclopedias, etc. (Kacprowicz & Borowiecka, 2014)
Applications that offer a service either for free or part of the money
collected goes to a social cause. For example free tours in the cities.
Doing good things or volunteering work and get as return time coins that
can be used in other services.
Table 5: Social Innovation Project Categorization
45
Based on that categorization, a further and more personalized analysis can be made in some
of the categories that contain more than 3 results. In this case only the success of the way is
going to be presented and not if the specific way is known to the participated projects or
not. The number of the participating projects is not that big to have solid results, although
some preliminary observations can be made.
Crowdfunding
In this category almost all the introduced ways have been used by the participants at least
once. Although not all of them have been positively graded, the majority of the gamification
elements like award points, award prizes, the use challenges and contests were
characterized as not very or not at all useful. On the other hand ways from the online
community category were highly graded.
Figure 20: Crowdfunding: How successful are these ways to Increase User Engagement?
Community Tool
On the projects of the community tool category had more clear results with bare use of
gamification elements and high preference on ways belonging to the online communities
and social layer categories. All these ways had also better grade comparing with the other
project categories.
46
Figure 21: Community Tool: How successful are these ways to Increase User Engagement?
Civic Engagement
On the civic engagement project category, ways from the motivational factor and the online
communities’ category were the most used and the best rated by the social innovation
projects belonging in this category. Some attempts to use gamification elements have been
made but the number of projects that have tried it is not that enough to make a solid
observation.
Figure 22: Civic Engagement: How successful are these ways to Increase User Engagement?
47
Social Mapping
Ways from the General and Social layer categories were used from the majority of the social
mapping projects scoring also high grades.
Figure 23: Social Mapping: How successful are these ways to Increase User Engagement?
Sharing Economy
Projects from the category of Sharing Economy, did not showed any particular preferences
in some of the categories. Although giving feedback, usage of reminders and goals and
achievements are ways that have been rated highly.
Figure 24: Sharing Economy: How successful are these ways to Increase User Engagement?
48
The rest of the categories does not have enough participants to make some conclusions. For
that reason they are not depicted with graphs as the previous ones. That is one of the main
limitations of this research and part of the future work that is proposed in section 9.3.
By conducting this survey, a lot of insight from the real world were collected. The extracted
ways from the literature review has been evaluated and rated by the social innovation
projects that participated. All these data have been collected and used as a base for the
treatment design that is presented in the next chapter.
49
Chapter 7
Implementation of the user
engagement method repository
As it was mentioned in Chapter 2, one of the objectives of this research is the creation of an
automated system that will assist the social innovation project owners to select the bestfitted ways for their projects. To achieve that, a systematic literature review has been
conducted having resulting in24 ways to increase user engagement, which have been
analyzed in subchapter 5.4.2. These ways were evaluated using a survey by 39 social
innovation projects as it was presented in the previous chapter. Based on these survey
results, an automated system was build.
7.1.
Technology and design
The automated system 9 is a part of the existing website10 that was used to promote this
research. It was created using PHP for the connection with the database, HTML and CSS for
the formations of the results and MySQL for the database. In Figure 25, a high-level model of
the architecture of the system is presented.
Figure 25: Automated System Model
The user makes a request to the system asking for personalized results. The system is using
the data from the user request to filter the results and provide a personalized and bestfitted solution to the user. The user has the option to filter the general results with the type
9
http://boostingsocialinnovation.com/results.php
http://boostingsocialinnovation.com
10
50
of category as they have been presented in the Table 5. The user can also sort the results
according to the evaluation from the users, how popular are to the users (if it is widely
known or not) and if they are used. All the previous are calculated in percentages to make
comparisons easier. The functions used for the calculations are presented below.
𝑝𝑜𝑝𝑢𝑙𝑎𝑟𝑖𝑡𝑦 = 𝑟𝑜𝑢𝑛𝑑
𝑢𝑠𝑒𝑑 = 𝑟𝑜𝑢𝑛𝑑
𝑟𝑎𝑡𝑒 = 𝑟𝑜𝑢𝑛𝑑
#𝑝𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑛𝑡𝑠 − #𝑛𝑜𝑡𝑘𝑛𝑜𝑤𝑛
, 2 ∗ 100;
#𝑝𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑛𝑡𝑠
#𝑝𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑛𝑡𝑠 − #𝑛𝑜𝑡𝑘𝑛𝑜𝑤𝑛 + #𝑘𝑛𝑜𝑤𝑛
, 2 ∗ 100;
#𝑝𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑛𝑡𝑠
#𝑡𝑜𝑡𝑎𝑙𝑟𝑎𝑡𝑒
3 ∗ #𝑝𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑛𝑡𝑠 − #𝑛𝑜𝑡𝑘𝑛𝑜𝑤𝑛 + #𝑘𝑛𝑜𝑤𝑛
, 2 ∗ 100;
The number of participants (#participants) represents how many projects were evaluated in
a certain way. The general case (no filters applied) with the current data the number is equal
to 39. The number of #notknown, #known is the summary of the corresponding replies. The
number #totalrate is calculated by the following function.
#𝑛
#𝑡𝑜𝑡𝑎𝑙𝑟𝑎𝑡𝑒 =
0 ∗ Not at All] + 1 ∗ 𝑁𝑜𝑡 𝑉𝑒𝑟𝑦] + 2 ∗ 𝑆𝑜𝑚𝑒𝑤ℎ𝑎𝑡] + 3 ∗ 𝑉𝑒𝑟𝑦] ;
𝑖=0
In Figure 26, a beta version of the automated tool is presented.
Figure 26: Automated System - Beta Version
A description of the tool and its functionalities is placed on the top of the tool right beneath
the title of the page.
51
On the main panel, the ways to increase user engagement are presented having the
following data:
-
Classification of the way
Description and references
A value in percentage depending on the sort of methods that have been applied.
The way these percentages have been calculated have been explained before.
Type of the way [Method, Technique, Guideline]
Validation level [Ad-Hoc, Case Study, Experiment, None]
Availability of supporting Tools [ Module, Standalone Software, N/A]
On the left side, there is a filter panel, where the user can select to filter the results. By
applying the filters, the values take into consideration the responses from the project that
belong to the specific category. On the right corner just above the main panel there is a
sorting tool, with which the user can sort the results based on their popularity (in what
percentage of the participants the specific way was known), rating (what was the rating from
the participants), and use (in what percentage the users were using the specific way) a
bigger image of the sorting tool can be found in Figure 27.
Figure 27: Automated System: Sorting tool
52
7.2.
Online Repository
In this section the online repository structure is explained using an example of one of the 24
ways to increase user engagement. The full version of this online repository is going to be
implemented in a future research work. The literature review results are not sufficient and
analytic enough -as mentioned before- for every way to have its own completes description.
A future individual research specifically for each one of the 24 ways will be the ideal,
resulting in more accurate information about each way, tools available in the market at the
moment and the existence of models describing how to implement each way. Indicatively,
the usage of "levels" as a way to increase user engagement has the following record in the
online repository.
Name of the Way
Description
Example
Referred at
Type
Level of Validation
Supporting Tool
PDD
Use Levels
Levels are often defined as point thresholds, so that users can
automatically level up based on their participation, or use levels to indicate
status and control access to content on the site.
For example levels are different classes in frequent-flyer programs, colored
belts in martial arts, job titles in industry: an indication that you’ve
reached a milestone, a level of accomplishment in a community.
(Kumaresan, 2013)
Guidelines
None
N/A
Table 6: Online Repository: Use Level Record
53
Chapter 8
Treatment Validation
The last part of this research was to validate the proposed treatment. In other words, the
automated system that has been described in chapter 8, had to be tested with real users to
ensure the treatment fulfilled its purpose. This would also be an answer to the 4th research
question.
 RQ4: To what extent does the automated system serve its purposes of helping
problem stakeholders select the best-fitted methods or tools to increase user
engagement in their social innovation ICT projects?
The validation of the system was made using a questionnaire that can be found at appendix
12.7 and online at this link11. In this validation, 5 social innovation projects out of the total
39 were asked to fill out the questionnaire. More specifically, they were asked to evaluate
the tool rating the following sentences from “Strongly Agree” to “Strongly Disagree”:
-
The tool is clearly structured
The tool is interesting
The tool is relevant to my project
The tool deepened my understanding of the subject matter
The tool helped me to select the ways to increase user engagement for my project
The presentation of the ways is made in a very personalized way
I would recommend this tool to a friend who is working on a social innovation
project
Figure 28: Treatment Validation Questionnaire
11
https://goo.gl/forms/6kIjhPVq0NKGr6Pn2
54
The responses to the questionnaire show that the automated tool that has been created
fulfills its purpose. In detail, four out of five stated that agree or strongly agree that the tool
helped them to select the most proper ways to increase user engagement in their project.
All the participants rated highly both the design and the functionality of the tool. Moreover,
some of their positive comments were:
-
“It is easy to navigate and clear”
“It was straight to the point, very clear about its purpose and very helpful”
“Concise, good ranking, helped me think creatively and cut down on inefficient
things”
The only criterion that did not perform equally well in the evaluation process was the
personalization, meaning the way that the results were showcased to the users, within one
of the comments being:
-
“Would be better to have more details for each category, the sample size and the
project participants”
One more thing that the participants asked for, was to expand the research and add more
information for each of the presented ways and include also some examples of use for these
ways. That is something that was already taken into consideration during the design of the
online repository but due to time restriction could not be implemented for this version of
the tool. However, it is a highly prioritized goal for future implementation and it will be
further described in the section 10.3. More specifically, one of the participants claimed that:
-
“It would be nice, if each way could be linked with the more detailed description and
also if the three properties( Type, Validation, Supporting Tool) could also be
linked/explained”
55
Chapter 9
Discussion
This research is a first attempt to collect and evaluate all the ways to measure and increase
user engagement on social innovation ICT projects that are currently known while valuable
insights from the real world have been utilized to deepen the scientific knowledge in this
relatively new domain. An automated tool has been designed and implemented to assist the
social innovation project owners to select the best-fitted ways to measure and increase user
engagement, based on the data we have collected from the survey. The validation of the
tool showed that the proposed solution could help the social innovation stakeholders. A
collection of the main conclusions of this research is presented in section 9.1.
Of course not everything was easy and the author faced certain obstacles and limitations
during this research, a more detailed analysis which can be found in section 9.2. In addition,
although this research may have reached its end, I would like to believe that it would act as a
trigger for further research in the field of user engagement in social innovation projects. My
proposals for future work and further research are presented in section 9.3.
9.1.
Main Conclusions
The purpose of this research was to exhibit the multi-faceted problem of user engagement
in social innovation ICT projects and additionally propose an automated system in assistance
of the project owners. This system would assist them to select the best-fitted ways or tools
for them depending on their specified need. To do so, four research questions were
formulated and explored. In this section, the main conclusions of this research are
presented.
RQ1: How severe is the user engagement problem in Social Innovation ICT Projects?
RQ1.1: Are they any differences in the nature of the problems?
User engagement is a decisive factor in user retention, with the latter being one of the two
factors responsible for user growth and project success (the other one is user acquisition).
For social innovation projects in particular, user engagement is even more important
because most of the times the users participate voluntarily without receiving any material
compensation.
 User engagement is a crucial factor for the success of any project.
39 social innovation ICT projects from all around the world participated in the survey. The
analysis of their results showed that more than 70% of the projects face user engagement
problems from a moderate to a serious degree.
 User engagement problem is an existing problem for the Social Innovation ICT
projects.
56
From the survey analysis, there was not enough qualitative data to extract differences in the
nature of the problems. However some of the projects claimed that they faced the user
engagement problem when the buzz from the media stopped after their initial launch.
“When we first launched the project in August 2015, we received a lot of media attention.
Since then the participation rate has somewhat decreased.”
“The main problem is the low user retention. It started after the first month that the project
launched. I think the main reason is because users need more tools in order to come back
and stay on the website.”
RQ2: What existing methods are suitable to face the user engagement problem?
As it was mentioned before, to successfully face a problem, the state of the problem has to
be measured first. Then, the best possible ways to face the problem have to be selected and
applied accordingly, whereas the state of the problem has to be evaluated.
The analysis of the survey proved that the majority of the social innovation projects does not
measure properly user engagement, therefore making it difficult for the effectiveness of the
suggested ways to be measured.
 The Social Innovation ICT projects are not properly measuring User Engagement.
The systematic literature review that has been conducted revealed 12 ways to measure and
24 ways to increase user engagement.
 There are plenty of ways to measure and increase User Engagement in the
literature.
Although there are plenty of ways to measure and increase user engagement in the
literature, most of them are not widely known in the real world. According to the conducted
survey, about 50% of the ways were not used by 40% of the projects or more.
 In the real world, the aforementioned ways are far from widely known and
applied.
RQ3: Create a repository containing the best practices or tools that have been
collected and an automated system to assist the problem stakeholders to select the
best-fitted practices or tools to solve their problem.
An online repository has been created, containing all the data that have been extracted from
the systematic literature review. A better and more complete version has already been
designed and it will be implemented in a future research project.
An automated system has been designed and implemented having as its foundation the
aforementioned repository. This automated system is going to work as a tool for the social
innovation project stakeholders to select the best-fitted ways to increase user engagement
in their social innovation ICT projects.
57
RQ4: To what extent does the automated system serve its purposes of helping
problem stakeholders select the best-fitted methods or tools to increase user
engagement in their social innovation ICT projects?
The automated system has been validated by 5 social innovation projects showing that it
fulfills its purpose. Some of the comments claiming that are:
“Very thorough research, searchable and specific”
“It is informative and well done. Thank you! You saved me time!”
Of course there is space for improvement, something that it was also stated in the feedback
from the validation process and it has also been taken into consideration for future work.
Additional general conclusions that were revealed from this research are:
The validation of the ways to increase user engagement that is currently made from the
researchers should last longer in order to investigate the possible patterns from the
engagement and re-engagement cycles. As it was mentioned in sections 5.2 and 5.3, user
engagement is a factor that contributes towards a sustainable change of behavior. But in
order for it to have a measurable effect, validation periods should be longer.

The validation of the ways to increase user engagement should last longer
Last but not least, there are some insights regarding the current validation processes and
suggestions on how they could be improved. This is something that will enhance the
accuracy of the results about the actual effectiveness of the ways to increase user
engagement. Right now most of the case studies/experiments that have been used to
validate these ways, at least on the scientific work that has been studied by this research,
were using mockup or fake ICT platforms. This tactic could cause problems on the accurate
estimation and evaluation of the user engagement ways under examination. In section 5.2
and 5.4 the existence of a good ICT platform, has been characterized vital for the part of
triggering the user to perform an action as well as for sustainable user engagement.

9.2.
The validation of the ways to Increase user engagement should be made in real
projects.
Research Limitations
In this section the limitations of this research are going to be presented. This research was
part of my Master's thesis and that by itself had some basic time-related limitations and
others such as funding resources, research experience and language – geographical barriers.
This research had to be organized, designed and conducted in the pre-defined period of 8
months. This time-frame was a challenge from the beginning, the goals of the research as
well as the scope of it had to be determined based on that time-frame. Every action as they
have been defined in section 2, had a cost in time so even if sometimes it was tempting to
go further in the research or work more on the implementation of the solution in practice it
was not possible. As a result of the limited time the following tasks could have been
performed better:
58
 The survey that has been conducted, it only stayed online and was promoted
for a month.
A longer period of time, could make it possible to attract a bigger number of projects
to participate in the survey. That could be achieved either by just having more time
to insist on the “old school way”, that was described in section 7.4.1, either by
having the time to work with the inbound marketing strategies as they have been
described in section 7.4.2.
 The validation of the automated system has been conducted only via a survey.
Again a longer period of time would improve the participation ratio. Moreover
further validation using interviews and even better the physical presence of the
author with the questioned subjects during the testing, so that a better evaluation
of the responses and reactions could provide further insights
 The online repository has been constructed using only data from the general
literature review.
Due to the time limitation, the online repository has been constructed based only on
the scientific work that has been collected from the systematic literature review as it
was presented in section 4. Ideally a further research, maybe with a smaller
literature review, on each one of the ways to increase user engagement that is now
hosted in the online repository should have to be done.
 The resources to attract participants to the research were restricted.
The resources to attract participants to the research were restricted to the usage of
the network of impact hub and other incubators of social innovation, with most of
them being in Europe and speaking English. The language barrier had an effect on
finding and contacting projects based in East Europe, Asia and Latin America.
Moreover the existence of funding could assist the promotion of the research
through the advertising channels of Facebook and Linkedin.
 Lack of research experience
The last thing from the list of limitations is my lack of experience in the field of the
scientific research. That was my first big research, and of course there were
moments of uncertainty and periods of learning that at least cost me only time-wise
and not on the quality of the research.
59
9.3.
Future Work
Due to the novelty of this research domain, a lot of new directions for further exploration
have been revealed. The most interesting of them are going to be presented in this section,
as a call by the author to the scientific community to participate in the continuation of this
research on social innovation and user engagement.
 Continuation of the survey in order to collect more data giving the
opportunity for further analysis on the results.
The survey should be continued in order for more data to be collected. This would
certainly enhance the accuracy of the results and would enable further analysis of
the data. That can be achieved by following a two-dimensional approach to all the
data collected, making the most of the preliminary version of the automated tool
that has already been created.
The tool and this accompanying report could be distributed to the organizations that
work on the domain of social innovation as insights for the field, asking them in
return to encourage the projects they work with to fill out the questionnaire.
Secondly a more aggressive strategy on the approach of social innovation ICT
projects using the inbound marketing ways as it has been described in section 6.4.2
using the automated system as bait.
For both cases, a new function can be used in the automated system to encourage
the projects that have not participated in the survey to do so. That can be achieved
by displaying only the first 5 results of the tool and if the user wants to access all of
them, they will have to fill out the survey.
 In-depth research of the collected ways collecting more information to
complete the online repository and improve the functionality of the
automated tool.
As it was mentioned in section 7.2 and 0, further research on each one of the ways
to increase user engagement would improve dramatically the already existing
knowledge acquired by this research. A literature review for each one of the ways
would be the ideal way to collect the most accurate data to complete the online
repository. Having all these data would enable the automated tool to provide more
information to the users, by providing a page with all the extra information for each
one of the ways for increasing user engagement.

Implementing the automated tool by applying the proposed ways to some
real projects, having the ability to measure the user engagement before and
after the implementation.
Implementing the proposed treatment of this research – the automated system – on
some real projects will confirm the validity of the tool and the successfulness of it,
by being put into use.
60
The ideal scenario would be to work with some new social innovation ICT projects
that would design their project in compliance with the proposed framework,
following the proposed ways for user engagement.
61
Chapter 10
References
Aboulafia, A., & Bannon, L. J. (2004). Understanding affect in design: an outline conceptual
framework. Theoretical Issues in Ergonomics Science, 5(1), 4–15.
http://doi.org/10.1080/1463922031000086708
Baghaei, N., Freyne, J., Kimani, S., Smith, G., Berkovsky, S., Bhandari, D., Paris, C. (2009). SOFA: an
online social network for engaging and motivating families to adopt a healthy lifestyle. In 21st
Annual Conference of the Australian Computer-Human Interaction Special Interest Group:
Design: Open 24/7 (pp. 269–272). New York: ACM. http://doi.org/10.1145/1738826.1738871
Banhawi, F., Mohamad Ali, N., & Mohd Judi, H. (2012). User engagement attributes and levels in
Facebook. Journal of Theoretical and Applied Information Technology, 41(1), 11–19. Retrieved
from http://www.jatit.org/volumes/Vol41No1/2Vol41No1.pdf
Batey, M., Bull, R., & Decorme, R. (2013). Living Labs: Successful user engagement on EnergyEfficiency
through participatory innovation. In The International Society for Professional Innovation
Management (ISPIM) (pp. 1–14). Manchester.
Brien, H. L. O., & Toms, E. G. (2010). The Development and Evaluation of a Survey to Measure User
Engagement. Journal of the American Society for Information Science, 61, 50–69.
http://doi.org/10.1002/asi
Bruns, A. (2014). Beyond the Producer/Consumer Divide: Key Principles of Produsage and
Opportunities for Innovation. In The New Development Paradigm: Education, Knowledge
Economy and Digital Futures (pp. 51–65).
Bunchball Inc. (2010). Gamification 101: An Introduction to the Use of Game Dynamics to Influence
Behavior. Bunchball White Paper, (October), 14. http://doi.org/10.1016/j.compedu.2012.12.020
Butler, B. S. (2001). Membership Size, Communication Activity, and Sustainability: A Resource-Based
Model of Online Social Structures. Information Systems Research, 12(4), 346–362.
http://doi.org/10.1287/isre.12.4.346.9703
Castri, R., De Luca, V., Lobsiger-Kägi, E., Moser, C., & Carabias, V. (2014). Favouring behavioural
change of households’ energy consumption through social media and cooperative play. Behave
Energy Conference. Retrieved from
http://repository.supsi.ch/5541/1/behave14_social_power.pdf
Chang, T.-R., Kaasinen, E., & Kaipainen, K. (2012). What Influences Users’ Decisions to Take Apps into
Use?: A Framework for Evaluating Persuasive and Engaging Design in Mobile Apps for Wellbeing. International Conference on Mobile and Ubiquitous Multimedia (MUM), 2:1-2:10.
http://doi.org/10.1145/2406367.2406370
Christakis, N. (2008). Social Networks are Like the Eye. Retrieved from
http://www.edge.org/3rd_culture/christakis08/christakis08_index.html
Civic Engagement. (n.d.). Retrieved from http://www.apa.org/education/undergrad/civicengagement.aspx
Coursaris, C. K., & Van Osch, W. (2016). A Cognitive-Affective Model of Perceived User Satisfaction
(CAMPUS): The complementary effects and interdependence of usability and aesthetics in IS
design. Information and Management, 53(2), 252–264. http://doi.org/10.1016/j.im.2015.10.003
62
Coursaris, C. K., & van Osch, W. (2016). A Cognitive-Affective Model of Perceived User Satisfaction
(CAMPUS): The complementary effects and interdependence of usability and aesthetics in IS
design. Information & Management, 53(2), 252–264. http://doi.org/10.1016/j.im.2015.10.003
Cramer, H., Ahmet, Z., Rost, M., & Holmquist, L. E. (2011). Gamification and location-sharing: some
emerging social conflicts. Proceedings of ACM CHI Workshop on Gamification, 1–4.
Crowdfunding. (n.d.). Retrieved August 21, 2016, from https://en.wikipedia.org/wiki/Crowdfunding
Den Ouden, E. (2012). Meaningful Innovation. In Innovation Design: Creating Value for People,
Organizations and Society (pp. 61–96). inbook, London: Springer London.
http://doi.org/10.1007/978-1-4471-2268-5_4
Dupret, G., & Lalmas, M. (2013). Absence time and user engagement: evaluating ranking functions.
WSDM ’13: Proceedings of the Sixth ACM International Conference on Web Search and Data
Mining, 173–182. http://doi.org/10.1145/2433396.2433418
Emerson, M. (2011). Embedding BI into Your Software Solution - Best Practices.
Emmanuel, A., Scharf, K., & Szabadi, J. (2014). Studyroom: An Application for student collaboration. In
Canadian Engineering Education Association (CEEA14) Conference.
Farzan, R., DiMicco, J. M., Millen, D. R., Brownholtz, B., Geyer, W., & Dugan, C. (2008). Results from
Deploying a Participation Incentive Mechanism within the Enterpris. Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (CHI’08), 563–572.
http://doi.org/10.1145/1357054.1357145
Fitz-Walter, Z., Tjondronegoro, D., & Wyeth, P. (2011). Orientation Passport: using gamification to
engage university students. In Proceedings of the 23rd Australian Computer-Human Interaction
Conference on - OzCHI ’11 (pp. 122–125). http://doi.org/10.1145/2071536.2071554
Fogg, B. (2009). A behavior model for persuasive design. Proceedings of the 4th International
Conference on Persuasive Technology - Persuasive ’09, 1.
http://doi.org/10.1145/1541948.1541999
Geelen, D., Reinders, A., & Keyson, D. (2013). Empowering the end-user in smart grids:
Recommendations for the design of products and services. Energy Policy, 61, 151–161.
http://doi.org/10.1016/j.enpol.2013.05.107
Goldbach, K., & Gölz, S. (2015). Shaping new opportunities for smart energy efficiency services by
engaging users. In 10th Conference on Sustainable Development of Energy, Water and
Environment Systems. Dubrovnik.
Hamari, J. (2013). Transforming homo economicus into homo ludens: A field experiment on
gamification in a utilitarian peer-to-peer trading service. Electronic Commerce Research and
Applications, 12(4), 236–245. http://doi.org/10.1016/j.elerap.2013.01.004
Haren, V. (2012). Archimate 2.0 Specification. Berkshire: The Open Group.
Jacques, R. D. (1996). The nature of engagement and its role in hypermedia evaluation and design.
South Bank University.
Jennings, M. (2000). Theory and models for creating engaging and immersive ecommerce Websites.
SIGCPR ’00: Proceedings of the 2000 ACM SIGCPR Conference on Computer Personnel Research,
77–85. http://doi.org/10.1145/333334.333358
Julita. (2011). Difference Between Customer and Consumer | Difference Between. Retrieved August
5, 2016, from http://www.differencebetween.net/language/difference-between-customer-andconsumer/
Kacprowicz, M., & Borowiecka, B. (2014). E-volunteering Handbook.
63
Kamal, N., Fels, S., McGrenere, J., & Nance, K. (2013). Helping Me Helping You: Designing to Influence
Health Behaviour through Social Connections. In P. Kotzé, G. Marsden, G. Lindgaard, J. Wesson,
& M. Winckler (Eds.), Human-Computer Interaction -- INTERACT 2013: 14th IFIP TC 13
International Conference, Cape Town, South Africa, September 2-6, 2013, Proceedings, Part III
(pp. 708–725). inbook, Berlin, Heidelberg: Springer Berlin Heidelberg.
http://doi.org/10.1007/978-3-642-40477-1_49
Kitchenham, B. (2004). Procedures for performing systematic reviews. Keele, UK, Keele University,
33(TR/SE-0401), 28. http://doi.org/10.1.1.122.3308
Kumaresan, A. (2013). Cocreation Value Platform Based on User’s Behaviour to Increase the User
Engagement. In The 8th International Conference on Knowledge Management in Organizations:
Social and Big Data Computing for Knowledge Management (p. 643). Retrieved from
https://books.google.com/books?id=7IzFBAAAQBAJ&pgis=1
Kwon, K. H., Halavais, A., & Havener, S. (2015). Tweeting badges: user motivations for displaying
achievement in publicly networked environments. Cyberpsychology, Behavior and Social
Networking, 18(2), 93–100. http://doi.org/10.1089/cyber.2014.0438
Lalmas, M., O’Brien, H., & Yom-Tov, E. (2014). Measuring User Engagement. Synthesis Lectures on
Information Concepts, Retrieval, and Services, 6(4), 1–132.
http://doi.org/10.2200/S00605ED1V01Y201410ICR038
Lam, B., Chan, Y. K., Whittle, J., & Binner, J. (2011). Voice Your View: An Inclusive Approach to Civic
Engagement. Proceedings {…}. Retrieved from
http://include11.kinetixevents.co.uk/rca/rca2011/paper_final/F440_2317.PDF
Lee, J. J., Marks, J., Kern, R., & Jordan-cooley, W. (2013). Greenify : Fostering Sustainable Communities
Via Gamification. In CHI (pp. 1497–1501). http://doi.org/10.1145/2468356.2468623
Lehmann, J., Lalmas, M., Yom-Tov, E., & Dupret, G. (2012). Models of User Engagement. In J.
Masthoff, B. Mobasher, M. C. Desmarais, & R. Nkambou (Eds.), User Modeling, Adaptation, and
Personalization: 20th International Conference, UMAP 2012, Montreal, Canada, July 16-20,
2012. Proceedings (pp. 164–175). inbook, Berlin, Heidelberg: Springer Berlin Heidelberg.
http://doi.org/10.1007/978-3-642-31454-4_14
Liaqat, S., & Bhatti, Z. (2015). An Empirical Investigation Of How To Increase User Engagement On
Facebook Business Pages. In Southern Association for Information Systems Conference (p. 40).
Lieberman, M. (2012). Old School Advertising VS A New Inbound Marketing Strategy. Retrieved
August 20, 2016, from http://www.square2marketing.com/blog/bid/128207/Old-SchoolAdvertising-vs-A-New-Inbound-Marketing-Strategy
Lin, J. (2014). Toward increasing user engagement for mobile data collection applications using social
influence (University of British Columbia. EECE 496). Vancouver.
López, C., Farzan, R., & Brusilovsky, P. (2012). Personalized incremental users’ engagement: driving
contributions one step forward. In 17th ACM international conference on Supporting group
work (pp. 189–198). Sanibe Island, Florida, USA. http://doi.org/10.1145/2389176.2389206
MacDonald, S. (2013). Engaging the future. In Crossover - Collaborating for Success: (p. 12). Taiwan.
Retrieved from http://openair.rgu.ac.uk
Marczewski, A. (2013). Gamification: A Simple Introduction & a Bit More. http://doi.org/853508828
Michal, Ď. (2015). Gamification in knowledge management systems. Central European Journal of
Management, 1(2), 10. http://doi.org/http://dx.doi.org/10.5817/CEJM2014-2-3
Micheletti, A. (2016). Motivating and Involving Users through Gamification: A Proposal. Padua.
64
Miller, A. S., Cafazzo, J. a, & Seto, E. (2014). A game plan: Gamification design principles in mHealth
applications for chronic disease management. Health Informatics Journal, 1–10.
http://doi.org/10.1177/1460458214537511
Mulgan, G. (2012). The Theoretical Foundations of Social Innovation. In A. Nicholls & A. Murdock
(Eds.), Social Innovation: Blurring Boundaries to Reconfigure Markets (pp. 33–65). inbook,
London: Palgrave Macmillan UK. http://doi.org/10.1057/9780230367098_2
Mulgan, G., Tucker, S., Ali, R., & Sanders, B. (2007). Social Innovation: What it is, why it matters and
how it can be accelerated.
Murray, R., Caulier-Grice, J., & Mulgan, G. (2010). The Open Book of Social Innovation. Young (Vol. 30).
http://doi.org/10.1371/journal.pcbi.0030166
Nielsen, J. (2006). Participation inequality: Encouraging more users to contribute. Jakob Nielsen’s
Alertbox, 1–5. Retrieved from
http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:Participation+Inequality:+Enc
ouraging+more+users+to+contribute#0
Niragira, I., & Adeyeye, M. (2013). Incorporation of Gamification in a Localisation Platform for Usage
Optimisation. Cape Town.
O’Brien, H. L., & Toms, E. G. (2008). What is user enagement? A Conceptual Framework for defining
user engagement with technology. Journal of the American Society for Information Science and
Technology, 59(6), 938–955. http://doi.org/10.1002/asi.20801.1
Oh, J., Bellur, S., & Sundar, S. S. (2015). Clicking, Assessing, Immersing, and Sharing: An Empirical
Model of User Engagement with Interactive Media . Communication Research . JOUR.
http://doi.org/10.1177/0093650215600493
Oinas-Kukkonen, H., & Harjumaa, M. (2008). A systematic framework for designing and evaluating
persuasive systems. In Lecture Notes in Computer Science (including subseries Lecture Notes in
Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 5033 LNCS, pp. 164–176).
http://doi.org/10.1007/978-3-540-68504-3-15
Paulo, R., & Barreto, A. M. (n.d.). Social network sites’ content impact on user brand engagement: A
literature review.
Pfleeger, S. L., & Kitchenham, B. A. (2001). Principles of Survey Research: Part 1: Turning Lemons into
Lemonade. SIGSOFT Softw. Eng. Notes, 26(6), 16–18. article.
http://doi.org/10.1145/505532.505535
Phills, J., Deiglmeier, K., & Miller, D. (2008). Rediscovering social innovation. Stanford Social
Innovation Review, Fall, 34–43. http://doi.org/10.1111/j.1369-7625.2010.00656.x
PieterValkering, Erik Laes, Kris Kessels, M. U. and K. S. (2014). How To Engage End-Users In Smart
Energy Behavior? EPJ Web of Conferences 79, 04003 (2014), 4003, 1–8.
http://doi.org/10.1051/epjconf/20137904003
Ponciano, L., & Brasileiro, F. (2014). Finding Volunteers’ Engagement Profiles in Human Computation
for Citizen Science Projects. Human Computation, 1(2), 245–264.
http://doi.org/10.15346/hc.v1i2.12
Rampoldi-Hnilo, L., & Snyder, M. (2013). The business love triangle- smartphones, gamification, and
social collaboration. In Lecture Notes in Computer Science (including subseries Lecture Notes in
Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 8005 LNCS, pp. 309–315).
http://doi.org/10.1007/978-3-642-39262-7_35
Schor, J. (2014). Debating the Sharing Economy. Essay, Great Transition Initiative. Retrieved from
http://greattransition.org/publication/debating-the-sharing-economy
65
Stone, D., Jarrett, C., Woodroffe, M., & Shailey, M. (2005). User Interface Design and Evaluation (1st
ed.). Morgan Kaufmann.
Tarnawska, K., & Ćwiklicki, M. (2012). Evaluation of the European Social Fund as a Measure to
Support Social Innovation in the Public Sector. Economics and Management, 17(1), 237–243.
Tomayess, I., & Pedro, I. (2014). Human Computer Interaction and Usability in the New Participative
Methodology for Marketing Websites. Pacific Asia Journal of the Association for the Information
Systems, 6(3).
Tushman, M. L. (1977). Special boundary roles in the innovation process. Administrative Science
Quarterly, 22(4), 587–605. http://doi.org/10.2307/2392402
van de Weerd, I., & Brinkkemper, S. (2008). Meta-Modeling for Situational Analysis and Design
Methods. Handbook of Research on Modern Systems Analysis and Design Technologies and
Applications, 35–54. http://doi.org/10.4018/978-1-59904-887-1
Van De Weerd, I., Brinkkemper, S., Souer, J., & Versendaal, J. (2006). A situational implementation
method for web-based content management system-applications: method engineering and
validation in practice. Software Process Improvement and Practice, 11(5), 521–538.
http://doi.org/10.1002/spip.294
Vila, S. (2014). User engagement strategies for open data.
Webster, J., & Ho, H. (1997). Audience Engagement in Multimedia Presentations. SIGMIS Database,
28(2), 63–77. article. http://doi.org/10.1145/264701.264706
Wieringa, R. (2014). Design science methodology for information systems and software engineering.
Springer.
66
Chapter 11
Appendix
11.1. Ways to Increase User Engagement: Characteristics
Type
Supporting Tool
Level of Validation
Papers ids Method Technique Guideline None Module Standalone Ad-hoc Case Study Experiment None
Engage boundary spanners of
the targeted communities
Involve users in creating the
onlines communities
Engage users on the media
platform they are already
using
Provide analytics to the users
Maximize control on the ICT
project for the user
Provide Monetary Rewards /
Discounts
Give responsibilities to the
users
Set Goals and Achievements
Use Reminders
Give Feedback
Award Badges
Award Points
Use Rankings and
Leaderboards
Give In-App Awards and Prizes
Use Challenges and Contests
Use of Levels
Usage of virtual goods
Social Comparison
Use the Social Networks
Use Up-votes
Use Likes
Increase user's reputation by
their contribution
Enable Communication among
end-users
Use a combination of low and
high tech platforms
5
37
1
1
1
1
1
1
35
1
1
50, 54
2
2
1
54, 18
2
2
2
48
1
1
54
1
1
1
54, 52, 18
54, 43
61, 54, 35,
1
104, 78,
52, 44, 40
104, 44,
40, 49
104, 44,
40, 49, 52
3
2
3
2
2
1
4
4
1
4
4
1
3
4
2
1
4
5
2
1
1
2
3
1
2
1
2
1
1
1
44
104, 44,
49
49
49, 48
42, 61,
115
20, 104
48
48
1
1
2
1
2
3
1
1
1
1
2
1
1
1
1
1
1
2
1
2
1
1
1
1
1
1
2
1
2
1
1
2
1
1
1
1
88
1
1
1
13
1
1
1
35
1
1
1
11.2. Ways to Measure User Engagement: Characteristics
Type
Supporting Tool
Level of Validation
Papers ids Method Technique Guideline None Module Standalone Ad-hoc Case Study Experiment None
Interviews
Think aloud
Questionnaires
Mouse Tracking
Eye Tracking
Psychophysiological Measures
Absence Time
Click Through Rate
Click Depth
Return Rate
Conversion Rate
Average Reading Time / Dwell
Time
69
69
69, 68
74,81
69
1
1
69
1
1
1
2
1
1
1
1
2
1
1
1
1
2
1
1
1
1
1
1
2
69
74
74, 69
69
69
69
2
2
1
1
1
2
2
1
1
1
1
1
1
1
67
11.3. Survey
68
69
70
71
72
73
11.4. Email template
74
11.5. Reminder email
11.6. Reminder email for those that have clicked the survey link
75
11.7. Treatment Validation Survey
76