Anti-Phishing Phil
The Design and Evaluation of a
Game That Teaches People Not to
Fall for Phish
S. Sheng, B. Maginien, P. Kumaraguru, A.
Acquisti, L. Cranor, J. Hong, E. Nunge
CMU Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
2
Phishing email
Subject: eBay: Urgent Notification From Billing Department
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
3
We regret to inform you that you eBay account could be
suspended if you don’t update your account information.
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
4
https://signin.ebay.com/ws/eBayISAPI.dll?SignIn&sid=veri
fy&co_partnerid=2&sidteid=0
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
5
What is phishing?
 Social
engineering attack
 Misrepresents
electronic identity
 Tricks
individuals into revealing
personal credentials
 Defrauds
users
Financial Services Technology Consortium. Understanding and countering the phishing threat: A financial service
industry perspective. 2005.
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
7
Countermeasures for
phishing

Silently eliminating the threat
• Regulatory & policy solutions
• Email filtering (SpamAssasin)

Warning users about the threat
• Toolbars (SpoofGuard, TrustBar)

Training users not to fall for attacks
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
8
Design Rationale

Security is a secondary task

Learning by doing

Fun and engaging

Better strategies
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
9
Anti-Phishing Phil
 Online
game
• http://cups.cs.cmu.edu/antiphishing_phil/
 Teaches
people how to protect
themselves from phishing attacks
• Identify phishing URLs
• Use web browser cues
• Find legitimate sites with search engines
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
10
More about the game

Four rounds
• Two minutes in each round
• Increasing difficulty

Eight URL “worms” in each round
• Four phishing and four legitimate URLs
• Users must correctly identify 6 out of 8 URLs to
advance

In-between round tutorials
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
17
User Study

Test participants’ ability to identify phishing web
sites before and after training
• 10 URLs before training, 10 after, randomized
• Up to 15 minutes of training

Training conditions:
• Web-based phishing education
• Tutorial
• Game

14 participants in each condition
• Screened out security experts
• Younger, college students
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
18
Results

No significant difference in false negatives
among the three groups

Game group had fewest false positives
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
21
The effects

Improvement could be due to
• Learning to distinguish legitimate from phish
• Raising suspicion about all web sites

Learning is better than raising suspicion
• Fewer false positives
• Will help people more in the long run
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
22
Conclusions

Used signal detection theory to measure
effects
• Existing training materials increased suspicion
with little learning
• Game did not raise suspicion but resulted in
players learning to distinguish legitimate from
phish
 In some cases a little more suspicion would have
helped

Game condition performed best overall!
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
23
Acknowledgements

Members of Supporting Trust Decision
research group

Members of CUPS lab
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
24
Play Anti-Phishing Phil:
http://cups.cs.cmu.edu/antiphishing_phil/
CMU Usable Privacy and Security
Laboratory
http://cups.cs.cmu.edu/
Falling for Phishing
False Negative Rate
Pre test
0.5
0.4
0.43
Post test
0.38
0.34
0.3
0.2
0.19
0.17
0.12
0.1
0
Existing training
materials
Tutorial
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Game
26
Misidentifying Legitimate Sites
Pre test
False Positive Rate
0.5
Post test
0.41
0.4
0.30
0.3
0.30
0.27
0.21
0.2
0.14
0.1
0
Existing training
material
Tutorial
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Game
27
Lessons Learned

Pilot test
• Users be able to identify phishing
• But they misidentify real ones

Users tend to get the specifics, but not the
underlying concepts
• Conceptual – procedural knowledge

User didn’t ask father for help too much
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
28