QEST — 8 September 2014
PROBABILISTIC MODEL CHECKING OF
DTMC MODELS OF USER ACTIVITY
PATTERNS
Oana Andrei, Muffy Calder, Matthew Higgs, Mark Girolami
School of Computing Science, !
University of Glasgow, UK
Department of Statistics, !
University of Warwick, UK
Context and motivation
Mobile app developers vs. mobile app users
• How is my app used? Is the app used
as I intended? Are there distinct styles
of usage? If yes, what features make
them different?
• How is it used on cellular data vs. WiFi?
On different timezones or time of the
day? On different platforms?
• How could I change the app
(interface) to make it “easier” or more
“efficient” to use for a particular class
of usage? etc.
Oana Andrei | QEST 2014 | Florence, Italy
• Am I a typical user?
• How are my friends using it?
• I hope the next update will change
the interface to make it easier/faster
for me to get to the Settings menu…
but I don’t want to write feedback to
developers.
http://www.softwarepopulations.com
2
Our approach
Understanding mobile software usage
• Systematic and automated logging of
app usage — SGLog data logging
infrastructure: method calls and
contextual information
• Analyse logged data to asses design
problems and opportunities
• Provide insight into app usage,
including differences and similarities
between different usage styles and
populations of users
• Tailor apps to suit behaviours of actual
populations of users
• Iterative and incremental design for
appropriation, for improving user
experience and user engagement
Oana Andrei | QEST 2014 | Florence, Italy
Design assets
Development
Program code
Deployment
User-configurable system
Logging of use
Qualitative and quantitative data
Statistical inference
Categorised patterns of use
Deduction and property analysis
http://www.softwarepopulations.com
3
Statistical inference at work
Probabilistic Models of User Behaviour
!
• Choose a vocabulary of in-app
actions occurring in the logged user
traces.
• Choose an integer K > 0.
• Use the Expectation-Maximisation
algorithm to infer K activity patterns
(usage styles).
• Each activity pattern is a discrete-time
Markov chain with states labelled by
in-app actions.
• For each user learn its strategy over
the set of activity patterns. Define the
User Metamodel as a convex
combination of activity patterns.
Oana Andrei | QEST 2014 | Florence, Italy
!
!
EM estimates:
- K activity patterns α1, α2,…, αK
- for each user m, a vector of
mixture-weights θm = (θm(1), θm(2),
…, θm(K))
!
State transition in a User Metamodel:
In a state (s,i), a user m chooses an
activity pattern αj with probability
θm(j) and then moves to the next
state according to the probability
distribution Pj(s,·):
Pm((s,i),(t,j))= θm(j)·Pj(s,t)
http://www.softwarepopulations.com
4
Case study
Hungry Yoshi: a mobile, multiplayer game
• iOS app released in 2009 as a
research project at Univ. of Glasgow
• 320k downloads until now
• Based upon WiFi infrastructure
• The iPhone scans available WiFi
access points: password-protected
ones are shown as yoshi creatures and
non-password-protected ones as fruit
plantations.
• Each user has a basket which can
carry up to five pieces of fruit.
• The player should load their basket
with fruit from plantations, carry the
fruit to yoshis with a matching
appetite and feed the yoshis in
exchange for points.
Oana Andrei | QEST 2014 | Florence, Italy
http://www.softwarepopulations.com
5
Case study
Hungry Yoshi: a mobile, multiplayer game
plantations
basket
yoshis
Oana Andrei | QEST 2014 | Florence, Italy
• iOS app released in 2009 as a
research project at Univ. of Glasgow
• 320k downloads until now
• Based upon WiFi infrastructure
• The iPhone scans available WiFi
access points: password-protected
ones are shown as yoshi creatures and
non-password-protected ones as fruit
plantations.
• Each user has a basket which can
carry up to five pieces of fruit.
• The player should load their basket
with fruit from plantations, carry the
fruit to yoshis with a matching
appetite and feed the yoshis in
exchange for points.
http://www.softwarepopulations.com
6
Case study
Hungry Yoshi: screenshots
Main menu
Oana Andrei | QEST 2014 | Florence, Italy
Yoshi Zoe
Plantation NH
http://www.softwarepopulations.com
7
Case study
Hungry Yoshi: activity patterns for K=2
seeY
seeP
seeY
seeP
feed
pick
feed
pick
α1
α2
P(➞) > 0.1
P(➝) ∈ [0.01, 0.1]
P(⇢) < 10-12
Oana Andrei | QEST 2014 | Florence, Italy
http://www.softwarepopulations.com
8
Case study
Hungry Yoshi: intuition for activity patterns
seeY
seeP
seeY
seeP
feed
pick
feed
pick
Expert
behaviour?!
α1
Oana Andrei | QEST 2014 | Florence, Italy
α2
P(➞) > 0.1
P(➝) ∈ [0.01, 0.1]
P(⇢) < 10-12
Ineffective
behaviour?!
http://www.softwarepopulations.com
9
Case study
Hungry Yoshi: a user metamodel
θm(1)
seeY
seeP
feed
pick
α1
Oana Andrei | QEST 2014 | Florence, Italy
θm(2)
θm(2)
θm(1)
seeY
seeP
feed
pick
α2
http://www.softwarepopulations.com
10
Formal analysis of UMMs
Probabilistic Model Checking
!
• Formulate probabilistic temporal logic
properties in PCTL* to express
hypotheses about user behaviour and
relationship within and between
activity patterns.
• Formulate app-specific as well as
general questions posed by
developers.
• Probabilistic model checker PRISM
• Rewards/costs for performance
evaluation: expected score before
changing the playing style, expected
battery consumption within N button
taps for a particular playing strategy,
etc. (not in the paper)
Oana Andrei | QEST 2014 | Florence, Italy
!
Examples. What is the probability of
reaching a (goal) state: eventually, after
a number of in-app actions, after some
other event happens, etc.
!
- How many button taps does it take
to feed a yoshi for the first time?
- What is the probability of feeding a
yoshi for 5 times consecutively
without picking any fruit in
between? For each activity
patterns? Exhibit this behaviour
only after changing the activity
pattern.
http://www.softwarepopulations.com
11
Case study
Analysis of a Hungry Yoshi UMM for θ = (.7,.3)
What is the probability of feeding a yoshi for the first time within N button taps for
each activity pattern ?
!
P=?[(!"feed") U<=N (alpha=i)&("feed")]
α1
Oana Andrei | QEST 2014 | Florence, Italy
α2
http://www.softwarepopulations.com
12
Case study
Analysis of a Hungry Yoshi UMM for θ = (.7,.3)
What is the probability of feeding a yoshi for the first time within N button taps for
each activity pattern ?
!
P=?[(!"feed") U<=N (alpha=i)&("feed")]
α1
α2
Activity pattern α1 is more effective than α2 w.r.t. feeding a Yoshi rapidly.
Oana Andrei | QEST 2014 | Florence, Italy
http://www.softwarepopulations.com
13
Case study
Analysis of a Hungry Yoshi UMM for θ = (.7,.3)
What is the probability of feeding the same yoshi a full basket (five fruit in a row)?
!
P=?[F<=N((alpha=i)&"feed")] * pow(filter(state,P=?[X(alpha=i&"feed")],(alpha=i&"feed")),4)
Oana Andrei | QEST 2014 | Florence, Italy
http://www.softwarepopulations.com
14
Case study
Analysis of a Hungry Yoshi UMM for θ = (.7,.3)
What is the probability of starting with an activity pattern and not feeding a yoshi
within N button taps, then changing to the other activity pattern and eventually
first feeding a yoshi within N2 button taps?
!
P=?[(!(alpha=i)&!("feed")) U<=N (alpha=i&"feed")] * filter(state,P=?[(alpha=i)&!("feed") U<=N2 (alpha=i&"feed")], alpha=i&"feed") +
P=?[(!(alpha=i)&!("feed")) U<=N (alpha=i&"pick")] * filter(state,P=?[(alpha=i)&!("feed") U<=N2 (alpha=i&"feed")], alpha=i&"pick") +
P=?[(!(alpha=i)&!("feed")) U<=N (alpha=i&"seeY")] * filter(state,P=?[(alpha=i)&!("feed") U<=N2 (alpha=i&"feed")], alpha=i&"seeY") +
P=?[(!(alpha=i)&!("feed")) U<=N (alpha=i&"seeP")] * filter(state,P=?[(alpha=i&!"feed")U<=N2(alpha=i&"feed")],alpha=i&"seeP")
Oana Andrei | QEST 2014 | Florence, Italy
http://www.softwarepopulations.com
15
Case study
Analysis of a Hungry Yoshi UMM for θ = (.7,.3)
What is the probability of starting with an activity pattern and not feeding a yoshi
within N button taps, then changing to the other activity pattern and eventually
first feeding a yoshi within N2 button taps?
(α2,α1)
Oana Andrei | QEST 2014 | Florence, Italy
(α1,α2)
http://www.softwarepopulations.com
16
Case study: conclusions
Analysis of Hungry Yoshi usage: insights
• Activity pattern α1 corresponds to a more successful game playing strategy
than α2 and a user is much more likely to be effective if they change from α2
to α1 (rather than vice-versa), thus we conclude that α1 is expert behaviour
and α2 is ineffective behaviour.
• We also showed that the scenario where users were picking a full basket of
fruit and then feeding it all to a yoshi is quite rare (even for α1).
• As the model gets larger (the HY model has only 4 states) and we analyse
more user traces, we need automated formal analysis!
!
• These types of insights could inform a future redesign that helps users move
from ineffective to expert behaviour.
• More detailed models and analyses are needed to gain insights on the events
and configurations causing or preceding a change in the activity patterns
(such as reading the T&C or instructions page again).
Oana Andrei | QEST 2014 | Florence, Italy
http://www.softwarepopulations.com
17
Almost finished…
Open questions
• Can we guide the choice of K? We could start with an estimate for K and then
compare the activity patterns w.r.t. some probabilistic temporal properties: the
closer the results, the more similar are the activity patterns.
!
• What to log and what to/can we model from the logs? It depends on upon
the hypotheses or assumptions developers provide us with for analysis. E.g. for
Hungry Yoshi we need more info about T&C, seeding, gambling, etc.
!
• Formulating hypotheses: domain specific and generic. Are the specification
patterns for probabilistic quality properties of L. Grunske sufficient? Do we
need more complex patterns for the domain of mobile analytics?
Oana Andrei | QEST 2014 | Florence, Italy
http://www.softwarepopulations.com
18
Wrapping up
Conclusions and future work
What we are doing now:
• Software analytics for user interactive systems
• Probabilistic modelling and reasoning over actual user behaviours, based on
systematic and automated logging
• Real life case studies with thousands of users world-wide
!
What we are doing next:
• Longitudinal analysis — compare the first 1/4/8/24 weeks
• More detailed models and analysis for the Hungry Yoshi app and other apps
• Test other statistical models (e.g., DBNs, HHMMs)
• How to use the insights gained through analysis for future redesign
• Tool support
Oana Andrei | QEST 2014 | Florence, Italy
http://www.softwarepopulations.com
19
Thank you!
THE END
QUESTIONS? COMMENTS?
http://www.softwarepopulations.com
Oana Andrei | QEST 2014 | Florence, Italy