Streaming Sensor Data from the Home
What does it all mean? How can it help you?
!
Holly%Jimison,%Misha%Pavel,%%
Xuan%“Sean”%Li,%Krissy%Mainello%
College%of%Computer%&%InformaAon%Science%
Bouve%College%of%Health%Sciences%
ConsorAum%on%Technology%for%ProacAve%Care%
Northeastern%University%
%
!
!
!
•  Funding'
–  Na&onal!Science!Founda&on!
–  Na&onal!Ins&tute!on!Aging!
–  Alzheimer’s!Associa&on!/!Intel!Company!
–  Na&onal!Ins&tute!on!Standards!&!Technology!
–  TEKES!(Finland!Government)!
•  No'conflicts'of'interest'
'
•  Collabora5ve'work'with'
–  Oregon!Health!&!Science!University!
–  University!of!California!at!Berkeley!
Scalable Approach to Delivering
Health Interventions to the Home
!  Sensors, algorithms, mobile communications for
lifestyle interventions
!  Remote, just-in-time, continuous care
!  Incorporate principles of health behavior change
!  Optimal use of lower cost personnel
!  Integrate family & informal caregivers into the health
care team (untapped resource)
!  Platform for testing sustained cognitive interventions
in the home
Modular Software for Multiple Protocols
!  Cognitive Exercise (computer game format)
!  Novelty exercise
!  Physical Exercise
!  Sleep Management
!  Socialization
!  Medication Management
!  Mood Management (depression)
Behavioral Markers = Continuous Monitoring & Computational Models
Home health based on unobtrusive, continuous monitoring
Models!to!Infer!Ac&vi&es!of!Daily!Living!
Pavel et al., IEEE Special Issue, in press
6
Activity Monitoring in the Home
Sensor Events
Private Home
Bedroom
Bathroom
Living Rm
Front Door
Kitchen
Hayes et al., www.orcatech.org
Hayes, ORCATECH 2007
Activity Monitoring in the Home
Sensor Events
Residential
Facility
Bedroom
Bathroom
Living Rm
Front Door
Kitchen
Hayes et al., www.orcatech.org
Hayes, ORCATECH 2007
Measuring!Gait!in!the!Home!
•  Unobtrusive'gait'measurement'in<home'with'passive'
infrared'(PIR)'sensors'<'Hagler,%et%al.,%IEEE%Trans%Biomed%Eng,%2010%
–  Four!restricted!view!PIR!sensors!
–  Measure!gait!velocity!whenever!a!
–  ! subjects!passes!through!the!!
–  ! “sensorVline”!
–  Deployed!for!the!Intelligent!!
–  !!!!Systems!for!Assessing!!
–  !!!!Aging!Changes!(ISAAC)!study!
–  200+!subjects!monitored!for!>!4!years!
9
Subject!1!
0.035
90
0.03
Stroke
Velocity (cm/s)
80
0.025
70
0.02
60
0.015
50
0.01
40
0.005
30
12/07
08/08
11/09
12/10
Time
Aus&n!et!al,!Sept!2011!V!EMBC!(Gait)!
10
Subject!2!
90
Velocity (cm/s)
0.05
CDR=0.5
and MCI
diagnosis
0.045
0.04
80
0.035
0.03
70
0.025
0.02
60
0.015
0.01
50
0.005
07/07
02/09
09/10
Time
Aus&n!et!al,!Sept!2011!V!EMBC!(Gait)!
11
Health!Coaching!Pla_orm!
Family Interface
•  Safety
monitoring
•  Soft alerts
• Team-based
care
•  Socialization
Automated!Coaching!for!Physical!Exercise!
•  Collabora5on'with''
–  Oregon!Health!and!Science!University!
–  University!California!Berkeley!
•  Pre<recorded'video'clips'for'
tailored'exercise'and'Kinect'
Camera'
•  Real<5me'feedback'based'on'
image'interpreta5on'from'Kinect'
skeleton'representa5on'
•  Monitoring'of'balance,'flexibility,'
strength,'endurance'
•  Poten5al'for'remote'interac5on''
Sleep Module
Assessment
•  Sleep Hygiene
•  Anxiety
•  Circadian Rhythm
Tailored Intervention
14
Socialization Protocols for Cognitive Health
!  Web cams and Skype software given to participants
and their remote family partner
!  Frequent spontaneous use among participants
Cogni&on!V!Monitoring!&!Interven&on!
16
Computer!Game!to!Measure!Execu&ve!Func&on!!
17
Model!the!&ming!of!the!mouse!clicks!
Recall
Next Target
tR
Search for
Next Target
+ tS
( n, d )
Move to Next
Target
+
tM
S. Hagler et al., www.ORCATECH.org
18
Es&mates!from!Game!Predict!TMT!Scores!
R 2 = 0.78
p < 0.0001
S. Hagler et al., www.ORCATECH.org
19
Cognitive Modeling Example: Memory
E
D
C
B
B
E
D
C
C
B
E
D
C
F
B
E
D
BGHDG
F FGHD
BE FGH
EDE FG
D DE F
E
E
D
H
F
E
I
D
H
F
Characterize Memory Capacity
•  Intervening number of events
•  Intervening time
•  Memory load
Simple Memory Model: Discrete
Buffer
Probability of Correct
A
D
C
A
B
Probability of Correct
BACD
BAC
BA
B
Subject 1020, N = 8687
1
0.5
0
0
5
10
15
Intervening Number of Events
1
0.5
0
0
5
10
15
20
25
Intervening Time [sec]
Characterize Memory Capacity
with a Single Parameter
M Pavel, et al., www.ORCATECH.org
Dynamic!User!Model!to!Support!Tailored!Messaging!
Family Interface
•  Safety
monitoring
•  Soft alerts
• Team-based
care
•  Socialization
Family Caregiver Interface
Link to Demo
!!!!!!!!!!!!!!!!!!!!!!Monitoring!V>!Interven&on!!
•  Ac5vity'Monitoring'in'the'Home'
•  Cogni5ve'Monitoring'
–  Adap&ve!Computer!Games!–!Divided!Aaen&on,!Planning,!Memory,!
Verbal!Fluency,!+++!
–  Linguis&c!Complexity!–!Emails,!phone!
•  Motor'Speed'
–  Speed!of!Walking,!Computer!Typing,!Mouse!Movements!
•  Sleep'Monitoring'
•  Depression'–'affect'on'phone,'linguis5c'analysis'
•  Medica5on'Management'–'Context'aware'reminding'
•  Socializa5on'–'Skype,'phone,'emails'
•  Physical'Exercise'–'Interac5ve'video'
23