SurroundSense: Mobile Phone
Localization via Ambience
Fingerprinting
Written by Martin Azizyan, Ionut
Constandache, & Romit Choudhury
Presented by Craig McIlwee
Motivation
• Provide logical localization
• Using GPS only isn’t good enough
– Doesn’t work well indoors
– Doesn’t account for dividing walls
• Dedicated hardware is not scalable
Approach
• Create an ambience fingerprint using sound,
light, color, and user movement
– Noise signatures specific to type of location/store
– Chain stores have color themes
– User movement indicative of store type
Architecture/Algorithm
Architecture/Algorithm
• Data is recorded on the phone, preprocessed,
and sent to a server
• Filter module
– Subsets the candidates
– Wifi, movement, sound
• Match module
– Selects the best candidate
– Color/sound, Wifi
Architecture/Algorithm
• No single module needs to be perfect
– If each module is ‘good enough’ then all modules
combined are sufficient
– Being simple reasonably accurate instead of
sophisticated and perfect reduces resources
required for processing
Sound Module
• Filter
– Sound varies over time
• Fingerprints captured from various times of day
• Similarity of fingerprints is used to create a
threshold for a potential match
• Match if within the threshold, discard otherwise
– Threshold is generous
– More false positives is better than false negatives
Motion Module
• Filter
– Variations in user behavior
• Record 4 samples/second, use moving average
over last 10 samples
• Minor variations suppressed
Motion Module
• User movement is classified as stationary or
mobile
• 3 profiles defined
– Long stationary – restaurant
– Frequent movement with longer stationary –
browsing
– Frequent movement with shorter stationary –
shopping
• Some logical locations fit multiple profiles
Motion Module
Color/Light Module
• Match
• Images captured from camera while facing
downward
– Floor themes are consistent
– Other orientations introduce noise
– Common orientation when checking email, text
messages, etc
Color/Light Module
• Analyze patterns in the image
• First attempt was to convert pixels to RGB
values
– Failed due to shadow and reflection influences
• Second attempt was to convert to HSL values
– Isolates light on its own axis
Color/Light Module
Color/Light Module
• Same/similar colors result in clusters when
graphed
• Dominant colors generate larger clusters
• Similarity calculated as distance between
cluster centroids and size of the clusters
• Most similar candidate is the match
Wifi Module
• Normally a filter, match if camera is not
available
• Capture MAC address of available access
points every 5 seconds
• Compare occurrence ratio of currently
available access points to known access points
Known Issues
• Sound varies over time
– Split day into 2 hour windows, capture fingerprints
during each window
– No mention of day of week, time of year
• Camera in pocket
– All testing done with phone in hand
– Expected rise in wearable devices
• Mimicking user behavior
– Initial data showed artificial behavior
– Subsequent attempts shadowed real customers
Known Issues
• Resource (energy) intensive
• Accelerometer fingerprint takes time to
capture
• Non-business locations may not exhibit
enough diversity
– Offices, airports, libraries
Evaluation
• Recorded fingerprints of 51 locations
– “War-sensed” by students
– 2 different groups during different times of day
• Group A’s fingerprints used as database while
Group B was at the location collecting their own
fingerprints
• Accuracy analysis was done on various
combinations of sensors types
• All sensor types combined yielded 87% accuracy