Evaluating (Mental Health) Apps
John Torous MD
WISH/CHI, May 7th
Why Evaluate
• Increasing number of health apps
– 250,000+
– 10,000+ for mental health
• Many make bold claims
• Some are dangerous
• Some are useful
• Patients are using them right now
mHealth App Developer Economics 2016. Research2Guidance, October 2016.
http://research2guidance.com/r2g/r2g-mHealth-App-Developer-Economics-2016.pdf
@JohnTorousMD
App Stores Ratings are Not Helpful
K Singh et al. Many Mobile Health Apps Target High-Need, High-Cost
Populations, But Gaps Remain. Health Affairs. 2016
http://content.healthaffairs.org/content/35/12/2310.abstract
@JohnTorousMD
Expert App Ratings Have Low Inter-rater
Reliability
Powell AC, Torous J, Chan S, Raynor GS, Shwarts E, Shanahan M, Landman AB
Interrater Reliability of mHealth App Rating Measures: Analysis of Top Depression and Smoking
@JohnTorousMD
Cessation Apps. JMIR Mhealth Uhealth 2016;4(1):e15
A Dynamic Ecosystem
Larsen ME, Nicholas J, Christensen H. Quantifying App Store Dynamics: Longitudinal
Tracking of Mental Health Apps. JMIR Mhealth Uhealth 2016;4(3):e96
@JohnTorousMD
Need a Framework, Not Scores
No Static Score
Instead a hierarchy and
questions to guide informed
decision making, ensuring
relevant information is
considered.
@JohnTorousMD
APA App Evaluation Model
Safety
Efficacy
Usability
Interoperability
App
App
App
App
App
• Apps that
respect
privacy
and secure
data
• Apps that
have
evidence
to support
use
• Apps that
easy to
use and
stick with
• Apps that
share data
in useful
ways
• Remaining
Apps
@JohnTorousMD
Framework
@JohnTorousMD
Risk, Privacy, and Security Level
APA App Evaluation Model
Apps present some
unique risks that may
often be overlooked
including profiling,
loss of confidentiality,
and misinformation
@JohnTorousMD
Personal Data <-> Privacy
Torous J, Kiang MV, Lorme J, Onnela JP
New Tools for New Research in Psychiatry: A Scalable and Customizable Platform to Empower Data Driven Smartphone Research
JMIR Ment Health 2016;3(2):e16
@JohnTorousMD
Privacy: 1) Transparency
@JohnTorousMD
Privacy 2) Security
https://www.digitalhealth.net/2015/10/nhs-health-apps-library-to-close/
Evidence Level
APA App Evaluation Model
App developers often
make many claims
even though there is
currently little clinical
evidence to support
such. This does not
mean that apps don’t
work, but rather that
there is much we still
do not know.
@JohnTorousMD
Medical Evidence in Anxiety Apps
Firth J, Torous J, Nicholas J, Carney R, Rosenbaum S, Sarris J. Can smartphone
mental health interventions reduce symptoms of anxiety? A meta-analysis of randomized
controlled trials. Journal of Affective Disorders. 2017 Apr 25.
@JohnTorousMD
Ease of Use Level
APA App Evaluation Model
An app is only as
useful as you and your
patients find it to
actually use
Adherence
@JohnTorousMD
Adherence
Owen et al. mHealth in the Wild: Using Novel Data to Examine the Reach, Use, and
@JohnTorousMD
Impact of PTSD Coach. JMIR Mental Health. Dec 2015
Interoperability Level
APA App Evaluation Model
Apps should not
fragment care and the
patient and
psychiatrist should be
able to share and
discuss data or
feedback from the
app as appropriate.
@JohnTorousMD
Fragmenting Care?
https://www.healthit.gov/sites/default/files/DesigningConsumerCenteredTelehealtheVisit-ONC@JohnTorousMD
WHITEPAPER-2015V2edits.pdf
APA App Evaluation Model
Interoperability
Making sure data
is used
meaningfully
Ease of Use
Understanding
usability and
adherence
Evidence
Risk / Privacy /
Security
Assessing for
potential risk and
harm
Ground
Understanding
the context of the
app
Ensuring the app
may offer benefits
@JohnTorousMD
Pilot Session with n=25 Psychiatrists
10
9
8
7
6
Confidence in Using Apps
5
4
3
Pre/Post App Rating Score
out of 10
2
1
0
Pre Session
Post Session
@JohnTorousMD
Online at Psychiatry.org
@JohnTorousMD