CLOUDANDPERVASIVE
SYSTEMS
What is Ubiquitous Computing?
Ubiquitous
Ubiquitous (adjective) Everywhere
Noun: Ubiquity/Ubiquitousness, Adverb: Ubiquitously
From Wikipedia
Ubiquitous Computing
Ubiquitous Computing (Ubicomp) is a post-desktop
model of human-computer interaction in which
information processing has been thoroughly integrated
into everyday objects and activities.
Ubiquitous Computing engages many computational
devices and systems simultaneously, and may not
necessarily even be aware that they are doing so.
Video 1 about Ubiquitous Computing
Other Definitions
Ubiquitous Computing: Numerous, casually accessible, often invisible
computing devices, frequently mobile or embedded in the environment,
connected to an increasingly ubiquitous network infrastructure
composed of a wired core and wireless edges (NIST)
Ubiquitous Computing is when mobile phones, PDAs, pagers,
automobiles, refrigerators, and other easy-to-use devices are linked to
the Internet, allowing us to connect anytime, anywhere, a new
infrastructure that will be common, ubiquitous, and work invisibly. (IBM)
Physical Environments created when computing power and network
connectivity are embedded in everyday device and object at
everywhere in all time
Video 2 about Ubiquitous Computing
Who first proposed Ubiquitous Computing?
Weiser’s Vision (1990):
Ubiquitous Computing (UC, Ubicomp)
Mak Weiser’s
Mark Weither’s Ubicomp July/1999
Percomp
Pervasive
Industry Vision (1999, IBM, etc.):
Pervasive Computing (Percomp/Percom)
EU’s Vision (2001):
Ambient Intelligence (AmI)
Ambient
AmI
Ubiquitous Computing (UC, Ubicomp)
Physical Thing & Everyday Activity
In Real World on physical-cyber spaces in physical-digital form
e-Activity
Everyday Activity
Cyber World
Physical World
e-Thing
Physical Thing
WbS, SmW, Grid, P2P, EaaS, Cloud
UC, ID, Context, Emb. Sys., etc.
Computers & Networks/Internet
Sensor/M/NEMS, Comps & Per. Nets
Other Related Visions
Cyber Physical System (CPS) by US
Internet/Web of Things (IoT/WoT)
Smart World and Ubiquitous Intelligence by Ma
Smart Planet by IBM
U-Korea (from 2004/Nov)
U-Japan (from 2005)
Other Similar or Related Computing
Proactive Computing
Autonomic/Organic Computing
Context-aware Computing
Human Centric Computing
Embedded Computing
Wearable Computing
Sentient Computing
Sensor Network/Computing
Mobile Comp, Cloud Comp, Social Comp, ……
Ubicomp
very wide scope, related to many computing
Devices, Objects/Things Connected
The Economist, April 28, 2007
– RFID: Attachable/Buried –
(Radio Frequency IDentification)
- Embedded Small Computers -
BAS
ABC
ABS
ESP
ACC
ASC
Wearable Devices/Computers
More and more will appear!
Beyond Wearable: Paste-able, Combinable, Implantable
Paste-able
Combinable
Implantable
– Computer/Net Blended Textiles -
Elektex/IDEO flexible mobile phone
SoftSwitch qwerty-keyboard
Sensors and MEMS
Micro Electro
Mechanical System
Re-Checking Ubicomp Definitions
Mark Weiser: (Ubicomp is) a field on a physical world richly and
invisibly interwoven with sensors, actuators, displays, and
computational elements, embedded seamlessly in everyday objects of
lives and connected through a continuous network.
Ubiquitous Computing: Numerous, casually accessible, often invisible
computing devices, frequently mobile or embedded in the environment,
connected to an increasingly ubiquitous network infrastructure
composed of a wired core and wireless edges (NIST)
Physical Environments created when computing power and network
connectivity are embedded in everyday device and object at
everywhere in all time
Ubicomp
Ubiquitous Computers/Devices/Objects/Things
Ubiquitous Networks/Connection/Communications
Applications in Physical World, Everyday Lives
RFID-based Applications
Logistics Video
Implant V1
Implant V2
RFID-based Library Video
Sensor Network Applications
Precision Agriculture
Precision agriculture aims at
making cultural operations
more efficient, while reducing
environmental impact.
Information collected from
sensors is used to evaluate
optimum sowing density,
estimate fertilizers & others.
Sensor and Actuator
A sensor is a device that measures a physical quantity and
converts it into a signal which can be read by an observer
or by an instrument. (Wikipedia) Input Signal
Output Signal
Sensor
An actuator is a device for moving/controlling a mechanism/system,
or generate some output, e.g., motor, LED, buzzer, speaker, etc.
Sensors and actuators are bridges between real and digital worlds
Types of Sensors
Acoustic, sound, vibration
-- Microphone, geophone, seismometer, accelerometer, …
Automotive, transportation
-- Speedometer, map sensor, water sensor, parking sensor, …
Chemical
-- Sensing carbon, gas, hydrogen, oxygen, smoke, etc.
Electric, magnetic, radio
-- Hall effect, magnetometer, metal detector, telescope, …
Environment, weather, moisture, humidity
-- Leaf sensor, rain/snow gauge, pyranometer, …
Flow, fluid velocity
-- Air flow meter, flow sensor, water meter, …
Ionising radiation, subatomic particles
-- Cloud chamber, neutron detection, particle detector, …
Types of Sensors
(Cont.)
Navigation instruments
-- Air speed indicator, depth gauge, gyroscope, turn coordinate, …
Position, angle, displacement, distance, speed, acceleration
-- Accelerometer, position sensor, tilt sensor, ultrasonic sensor, …
Optical, light, imaging
-- Colorimeter, electro-optical sensor, infra-red sensor, photodiode, …
Pressure
-- Barometer, boost gauge, pressure gauge, tactile sensor, …
Force, density, level
-- Force gauge, level sensor, load cell, hydrometer, …
Thermal, heat, temperature
-- Heat sensor, radiometer, thermometer, thermistor, …
Proximity, presence
-- Motion detector, occupancy sensor, touch switch, ...
A list of various kinds of sensors in Wikipedia
Sensor Networks
A sensor network (SN) is consisted of multiple interconnected sensors.
A wireless sensor network (WSN) consists of spatially distributed
autonomous sensors (called sensor nodes) to cooperatively monitor
physical or environmental conditions
Sensors + Wireless Networks
Traditional Networks vs WSN
An Introduction Video about Wireless Sensor Networks
Sensor Nodes
A sensor node, also called a mote in North America, is a WSN node
that is capable of performing some processing, gathering sensory
information and communicating with other connected nodes in the WSN.
Examples for Wireless Sensor Nodes
Dot Mote
Rene Mote
MICA Mote
weC Mote
Typical Multiple WSN Architecture
A sensor network normally constitutes a wireless ad-hoc network,
and each sensor supports a multi-hop routing algorithm where
nodes function as forwarders, relaying data packets to a base station.
Multiple Sensor Networks System
Distribution field of
phenomena that
can be detected
measured
Physical
Phenomena
S
S
Sensor
net
S
S
S
Storage
S
S
S
Sensor net
Sensor
net
S
S
S
S
S
Sensors that
detect event
User
S
Access
Node
Sensors that
notify access
node
Internet
Sensor Data Management
sensors
observer
phenomena
Observer interested in phenomena
with certain tolerance
Accuracy, freshness, delay
Sensors sample the phenomena
Sensor Data Management
Determining spatio-temporal
sampling schedule
Difficult to determine locally
Data aggregation
Interaction with routing
Network/Resource limitations
Congestion management
Load balancing
QoS/Realtime scheduling
Sensing Information Fusion
One sensor is (usually) not enough
- Noisy, limited accuracy, unreliable – failure, environment restriction, etc.
Sensor fusion - Combine readings from several sensors into a better one
Obj
Sensor 1
Preprocessing
(1.10, 1.06)
1.08
Sensor 2
Preprocessing
(0.92, 0.96)
0.94
Sensor 3
Preprocessing
(0.30, 0.36)
0.33
Sensor N
Preprocessing
(0.94, 0.96)
Sensing
1.08+0.94+0.95
3
0.99
Fusion
Interpretation
0.33?
Ignore 0.43
0.95
Perception
Sensing Information Fusion
Preprocessing
- 'Cleanup’ the sensor readings before using them, e.g.,
noise reduction, re-calibration, unify the format, etc.
Data Fusion
-
Combine data from different sources
Measurements from different sensors
Measurements from different positions
Measurements from different times
Techniques that take into account uncertainties in data sources
- Discrete Bayesian methods
- Neural networks
- Kalman filtering
Sensor Web (Cont.)
What are the Internet?
The Internet … a Network of Networks that consists of millions of
private, public, academic, business, and government networks, of
local to global scope. - From Wikipedia
Originated from the ARPANET around 1970
Available from 1980, got popular from 1990.
Key components
- Hardware: Routers connecting networks
- Software: TCP/IP protocol suite, IPv4
IPv6
- Addressing: 2**32 (IPv4)
2**128 (IPv6)
- Naming: DNS
symbolic names
The Internet
Leonard Kleinrock
router
physical net
A Global Net
Connecting All
Computers
Internet of Computers (IoC)
Lawrence Roberts
Jon Postel
Steve Crocker
Vinton Cerf
Robert Kahn
What are the Web?
The World Wide Web, abbreviated as WWW or the Web, is a system
of interlinked documents accessed via the Internet. - From Wikipedia
The Web was originated from Tim Berners-Lee around 1990.
The Web, like Email, is one of the services that runs on the Internet.
Key components
- Uniform Resource Locator (URL) & Uniform Resource Identifier (URI)
- HyperText Markup Language (HTML)
1st Web
Graphical
- Hypertext Transfer Protocol (HTTP)
Browser
- Web server and web browser (client)
The Web
Ted Nelson
Hypertext
Internet of Documents (IoD)
Tim Berners-Lee
1st Web Server
Mark Andreesen
Netscape
How about Social Media/Web2.0?
SM/Web2.0
Internet of People (IoP)
How about Cloud Computing?
Software
Cloud
Internet of Resources (IoR)
What are Things?
Thing - An object, an entity, an idea, or a quality perceived, known, or
thought to have its own existence, … (dictionary)
Object – A tangible/visible thing; a person or thing seen as a focus or
target for feelings, thought, etc.; a purpose/objective; ... (dictionary)
Everyday Things/Objects – used in human daily lives
Inner Things – mind, directly insensible things, ...
Physical things, digital things, real/virtual things, …
Various Things!
Many IoX!
What Kinds of Things in IoT?
IoC
IoD
IoP
IoR
Examples of Things in IoT
Identification of a Thing
What is a thing?
We distinguish two classes of things
Things that are computers including smart phones equipped
with communication interfaces.
Things that are not computers, but who are associated with
computers equipped with communication interfaces.
What is the identifier of a thing?
They are several proposals:
A serial number, such as an EPC code.
An IP address, MAC address of Wifi, BlueTooth, SIM card code
Other, for example a fix hash value, or ad-hoc naming scheme.
Authentication
Is there a need/way to authenticate a thing?
In other words, is it possible and needed to prove the
identity of a thing?
IoT Platform and Applications
From CERP-IOT
Challenging Issues in IoT
How is a thing identification structured? (the object naming)
Who assigns the identifier to a thing? (the assigning authority)
How and where can additional information about that thing be
retrieved, including its history? (the addressing mechanism and the
information repository)
How is information security/privacy/trust/safety ensured?
Which stakeholders are accountable for each of the above questions,
what is the accountability mechanism?
Which ethical and legal framework applies to the different
stakeholders?
What are uniform thing naming scheme, communication protocols
between various things, thing’s data collection, storage, query,
management, processing, visualization, use, security, privacy, ….
IOT TODAY
• Proprietarydevices
• Stillsmall"intercommunication"
• Cloud-basedplatforms
• Alldatagoestothecloud
• Smallornolocalintelligence
• Problem:notallapplicationsfitthe"allinthecloud"model
• Proximityservices
• Latencyissues
• Privacyissues
42
IOT CHALLENGES
• IoT ismuchmorethanWSNs
• Complexdataflow
• Datasources
• Dataconsumers
• Reasoningservices
• Actuators
• Hownetworksmayaffect
theIoT experience
• Delays/latency
• Connectionissues
• Privacyconcerns
43
CURRENTLANDSCAPE
• ManyIoT applicationsusecloudservicesasdataaggregator
• MostIoT PaaSAPIs
• Focusondatagathering/visualization
• Havelimiteddataprocessingcapabilities
• Triggers,listeners
• IoT reasoningisdelegatedtoadifferentprocessingentity
• Latencyandprivacyissues
PRIVATECLOUDSANDPERVASIVEGRIDS
• Privateclouds
• Advantage:fullcontrolanddatalocality
• Drawback:dedicatedresources,complextodeploy
• Pervasivegrids
• Relyonalightweightmiddleware
• Executesoveravailableresources
•
Shouldadapttothenodescapabilities->heterogeneity
PROXIMITY CLOUDS OR"LASTMILE"FACILITIES?
• Satyanarayanan (2009)proposedadifferentviewofthe
cloud
• Cloudlets
• Cloudletsaimatrespondingtolatencyandprivacyissues
• Initialdataaggregation/filtering
• Dataanonymization /cryptography
• Providefastreactiontocriticalevents
• Cloudletssitbetweenthedevicesandthecloud,providing
partofthework
• Multi-scaleapproach
EDGE,FOG,EDGE CENTERED
• Recentworkstrytoapplytheconceptsofmulti-layeredcomputing
• Edge– proximitydevices(antennas,localservers)offersomeservices
• Mostlypromotedbytelecomoperators
• Fog– anydeviceispartoftheservice.Cloudeverywhere
• PromotedbyCisco
• Edge-centered– differentparadigm:thefocusmustbeintheedge(human
interaction)
Security
Security is the degree of protection against danger,
damage, loss, and criminal activity.
Security has to be compared to related concepts: safety,
continuity, reliability.
The key difference between security and reliability is that security
must take into account the actions of people attempting to cause
destruction.
IT Security Categories
Data Security
Information Security
Network Security
Computer Security
Application Security
…
General Security Requirements
Secure systems are often defined to fulfill three basic requirements,
also called the "CIA triad":
Confidentiality means that private data should only be
accessible to authorised users. It is sometimes also called secrecy.
Integrity means that it should be impossible to undetectably
modify protected data.
Availability means that authorised users should always (or at
least at clearly defined time periods) be able to access data or
services. The implication is that unauthorised users should be unable
to deny access for authorised users.
Types of Security Attacks
A
B
A
B
B
A
interruption
(denial of service)
interruption
interception
B
A
B
*Mobile code:
viruses, worms,
Trojan horses,…
modification,
eg, replay
fabrication,
eg, masquerade
Security Issues in Ubicomp
Wireless media supporting from personal-area to widearea networks
Ad-hoc device association at different layers
Location and context considerations in policy
management
Heterogeneity of content encoding
Variability in processing and storage capabilities of
devices
Heterogeneity of security & privacy policies
…
RFID-related Privacy Problem
Wig
Replacement hip
model #4456
medical part #459382
(cheap polyester)
Books and Their
Names
xxx, yyy, zzz
1500 USD
in wallet
30 items
of lingerie
Serial numbers:
597387,389473
…
Ubicomp: Physical-Cyber Loop & Consequence
Sensors
Physical Input
User
Object
Automatic
Phy Cyb Phy
Physical Output
Actuators
Loop
Consequence of
misbehavior or
unexpected output
from trustor/trustee?
Loss
Unsafe
Danger
User Protection
- Tech. + Social
Cyber Environments
Computers, Components
Software, Service, …
Trust
Safety
measure, model,
monitor, management,
analysis, evolution, …
Safety Guarantee
- Sec. + Rel. + …
Vulnerabilities of Cyber-Physical Systems
•
•
•
•
Controllers are computers
Networked
Commodity IT solutions
New functionalities (smart
infrastructures)
•
•
Many devices (sensor webs)
Highly skilled IT global
workforce
(creating attacks is easier)
•
Cybercrime
Smart Space/Environment
Smart Space/Environment
Computerized active one
- Physical environments but digitally enhanced/integrated
- Awareness of contexts: users, ambience, resources, etc.
- Context-Aware responses or services with certain intelligence
- Where are users?
Supervisors & computing elements!!
Ubi/Pervasive
Information
- Acquisition
- Archive
- Analysis
- Awareness
- Application
-…
Sensors
MM Information
Context
Loop
Actuators
Comfort Service
Context-Aware
Space
Environment
Object
Artifact
Plant
Body
(User)
Numbers of Smart City Projects
Europe
North
America
China
Far-East
India
Japan
Africa
Pacific
Ocean
South
America
Toyota Smart Center in 20xx
http://itpro.nikkeibp.co.jp/article/Keyword/20120906/421044/
As of September 12, 2012
Implications/Facts of Ubicomp
Invisible Computers/Devices Everywhere
7 Billions People x 1,000 = 7 Trillions (1012) computers/devices
Pervasive Networks/Communications/IoT Everywhere
7T x 150 Messages/day > 1015 messages (1P) per day
Information Sensing/Monitoring Everywhere
10T x 300KB/day x 365 > 1021 Bytes (106PB=103EB=1ZB) per year
“Smart/Intelligent” Things Everywhere
Zettabyte
Weiser’s words in his last days
“If the computational system is invisible as well as extensive, it becomes hard
to know what is controlling what, what is connected what, where information
is flowing, how it is being used, what is broken, what are the consequences of
any given action (including simply walking into a room). Maintaining
simplicity and control simultaneously is still one of the major open questions
facing ubiquitous computing research.”
- IBM System Journal, Vol. 38, No. 4, 1999.
Information Privacy
“The desire of people to choose freely under what circumstances
and to what extent they will expose themselves, their attitude and
their behavior to others.”
– Alan Westin, 1967
Privacy >> Secrecy!
- M. Langheinrich
“The problem, while often
couched in terms of privacy,
is really one of control.”
– M. Weiser & Brown, 1999
Internet Privacy & Cloud Privacy
Internet Privacy
Personal privacy concerning transactions/transmission of data via
the Internet – a subset of Computer Privacy.
HTTP cookie is data stored on a user's computer that assists in
automated access to websites or web features, or other state
information required in complex web sites. It may also be used
for user-tracking by storing special usage history data in a cookie.
The same for Flush cookies, Ever cookies, ISP, etc.
Social Networks (Facebook, etc.) keep track of all interactions
used on their sites and save them for later use.
Malware, Spyware, Web Bug, Phishing, …
Cloud Privacy
My data is in some “secure place” on the sky. Providers ensure all
critical data are masked and only accessible by authorized users.
Do we really have privacy when all our data on unknown clouds?
Ubicomp Privacy Implications & Challenges
Data Collection, Types & Access
Scale (everywhere, anytime)
Manner (inconspicuous, invisible)
Motivation (context!)
Observational instead of factual data
Internet of Things (IoT)
How to inform subjects about data collections?
Unobtrusive but noticeable
How to provide access to stored data?
Who has it? How much of this is “my data“?
How to ensure confidentiality, and authenticity?
Without alienating user!
How to minimize data collection?
What part of the “context“ is relevant?
How to obtain consent from data subjects?
Missing UIs? Do people understand implications
Green/Eco Ubicomp System
Ubicomp
huge number of devices
7 Trillions given
Computing/Com
how much energy in total?
Average Google Search
7g CO₂ (Alex), 0.2g CO₂
(Google searches can generate the same amount of CO2 as boiling a kettle)
Green Computing (Green IT)
refers to environmentally sustainable computing or IT (Wikipedia)
the study and practice of designing, manufacturing, using, and
disposing of computers, servers, and associated subsystems—
such as monitors, printers, storage devices, and networking and
communications systems—efficiently and effectively with minimal
or no impact on the environment. – by San Murugesan, 2008
Green Ubicomp Systems
Challenge ubiquitous Comp & Com!!
Power/Energy Aware System
Power/Energy Harvest System
e-waste
u-waste
CLOUDFIT
• PaaS computingframeworkbasedonP2Poverlays
• FollowstheFiniteIndependentIrregularTasks(FIIT)paradigm
• Designedforcomputingonpervasiveenvironments
• Faulttolerance
• Decentralizedscheduling
• DHTstoragewithreplication
• SimpleAPI
1.
Howmanytasks?
2.
Whatataskmustdo?
launcher
• Whichdatatoaccess,whichactions
• Optionaltaskdependency(DAGs)
receiver
CLOUDFIT FORIOT
• IoT servicesasCloudFIT applications
• Datastorage
VISUALISATION
SUBMIT
• Goals
COMMUNITY
CORE-ORB
• Localservices
• Pervasivenetworks
STORAGE ADAPTER
NETWORK ADAPTER
STORAGE / DHT
NETWORK OVERLAY
• Efficientdatahandling
IoT Devices
IoT App
Interface
CloudFIT nodes
3rd part
APP
Protocol
REST/
JSON
Network/
Others
• Computingpower
APPLICATION
CloudFIT
• Dataintegration
Service
WORKER
APPLICATION
Application
WORKER
63
MAPREDUCE ON PERVASIVEGRIDS
• Hadoop isthemostknownMRimplementation
• Notdesignedforheterogeneous/volatileenvironments
• Hierarchicalsolution(YARN/HDFS)withseveralissues
• Map-Reduceuseskey-valuepairs:(k,V)
• MRJobdefinedby2functions
• map: (k1;v1)→{(k2;v2)}
• reduce: (k2;{v2})→{(k3;v3)}
MAPREDUCE OVERCLOUDFIT
• Simpleexampletostresstheplatform
• IllustratestheprocessingofIoT informations
• Taskmanagement:
• Map-Reducephasesà 2consecutiveCloudFIT jobs
• Mapjob:asmanymaptasksasinputdatablocks
• Reducejob:oneormorereducersaccordingtothereducelogic
• CloudFIT nodescanleave/joinatease
• StorageandreplicationthroughtheDHT
• Fullydistributedscheduling
MAP-REDUCEOVERCLOUDFIT PERFORMANCES
500"
450"
Elapsed(Time((s)(
400"
350"
300"
CloudFIT"3"TomP2P"
250"
200"
CloudFIT"3"PAST"
150"
100"
Hadoop"
50"
0"
• et
0.5"GB"
1"GB"
Data(Volume((GBytes)(
2"GB"
AREALUSAGESCENARIO– OZONEWATCHER
• Since1985,theOzoneHoleisamajorindicatoroftheimpactof
humanactivitiesintheglobalecosystem
• CFC-likemoleculesdestroytheO3
• Reductiononthetotalcolumnozone(TCO)
• IncreaseofUVradiationonthesurface
• TostudytheOzoneHole,weneedtoexplore
ahugedataset
• Ozonesatellitereadingssince1978
• Aprox.1GBofrawdataeachyear
OZONESECONDARYEVENTS(OSE)
• OSEaredrasticreductionsontheozonecolumn
notrelatedtotheseasonalexpansionofthevortex
• AirmassespoorinOzonedetachedfromthevortex
• Mayreachmediumlatitudeswithahighpopulationdensity
Avg Nov 2015
4Nov 2015
ABIGDATAPROBLEM
• DetectingOSEisnotthathard,theproblemisthetiming
• Satellitedataispublishedwith1-2daysdelay
• Ground-basedequipmentcanenrichthedata
• WemustfindawaytoforecastOSE
• Goodforecastsimplytherecognitionofozonexwindspatterns
(trajectories)
• WinddatabasesalsorepresentsseveralGB
• Machinelearningtoextractcorrelations
HOWPERVASIVEGRIDSCANHELP?
• Computationalresourcesareapotentialproblem
• NotallcountriesconcernedbyOSEcankeepadedicatedclusteror
cloudleasing
• Thejobtypesmayvaryaccordingtotheapplication
• Longrunstolearncorrelationpatterns
• Shortrunstoperformdailydetection/forecast
• Pervasivegridsà usingavailableresourcesinanelastic
computationalgrid
OSEDETECTIONINCLOUDFIT
• Pipeliningofdifferentjobs
• Inputpreprocessing- transformtherawOMIdata
• Filteringandaggregation– selectdatacorrespondingtoageographicalzoneandtime
window
• Parameterscomputing- extracttimeseries,AVGandSTDEV
• Eventdetection- identificationofabnormalvalues
Parameters
OMI
Time Series
Time Series
OMI
Preprocess
Time
Series
Time
Series
avg
and
stdev
avg
and
stdev
OMI
Preprocess
Time
Series
Time
Series
avg
and
stdev
avg
and
stdev
OMI files
Preprocess
Filtering
and
Time
Series
avg and stdev
avg and stdev
UsingaPervasiveComputingEnvironmenttoIdentifySecondaryEffectsoftheAntarcticOzoneHole
Preprocess
Aggregation
avg and stdev
Preprocess
A.Steffenel,MKirschPinheiro,D.Kirsch-PinheiroandLVaz-Peres
DHT
Winds
Detection
Detection
Detection
Detection
Alert
DHT
INPUTPREPROCESSING
• TransformOMIdataintoexploitabledata
• Ourchoice:representeachdayinJSONdictionariestosimplifystorage(DHT
orNoSQL)anddataretrieval/filtering
L.A. Ste↵enel et al. / Procedia Computer Science 00 (2016) 000–000
Day:
1 Jan 1, 2013
OMI TO3
STD OZONE
GEN:13:003 Asc LECT: 01:44 pm
Longitudes: 360 bins centered on 179.5 W to 179.5 E
(1.00 degree steps)
Latitudes : 180 bins centered on 89.5 S to 89.5 N
(1.00 degree steps)
280280280280280280280280280280280280280280280280280280280279279279279279279
279279279279279279279279279279279279279279279279279279279279279279279279279
279279279279279279279279279279279279279279279279279279279279279279279279279
279279279279279279279279279279279279279279279279279279280279280280280280280
280280280280280280280280280280280280280280280280280280280280280280280280280
280280280280280280280280280280280280280281281281281281281281281281281281281
(...)
282282282282282282282282282282
lat = -89.5
(...)
lat = -88.5
(...)
lat = -87.5
(...)
(a)
{
"date":"20130101",
"step":"1.0",
"latitudes":{
"-89.5":["-179.5":"280","-178.5":"280",(...)],
"-88.5":["-179.5":"272","-178.5":"272",(...)],
(...)
}
}
(b)
Fig. 3. OMI Ozone raw datafile for a given day (a) and its JSON representation (b)
EVALUATINGDATASERIESFOROSEDETECTION
• AbasicmechanismtoidentifysuddendropsintheOzonecolumn
• Filterthetargetzoneandtimewindow(optional)
• ComputeAVGandSTDEVforagivenperiod(e.g.15days)
• Ifvalue<(AVG– 1.5xSTDEV)à ALERT
• Wecreateonecomputingtaskforeachdayand(X,Y)coordinate
• Paralleltasksdeployedinthepervasivegrid
VALIDATIONOFTHEAPPROACH
• Validationusingawell-knownOSE- October18th,2013
• Ozonedropped10%,withOSEreachingtropicallatitudes
• Also,theOSEanalysis
• Improvesthedetection
ofresidualOSE
• EvidencesOzonedrops
inotherglobeareas
ANOTEABOUTTHEPERFORMANCE
• Thepotential number oftasks is impressive
• 1-dayworldwide OSEdetection =64kcoordinates
• 30mininasmall clusterwith 24cores
• 2-3hours inaclusterwith 3Raspberry Pi
• OSE+Wind correlations shall multiply this complexity
• Winds aremultilayered (different altitudestoevaluate)
• Noworry!CloudFit inamulti-layered approach
• Clouddatabases fordatastorage
• Dedicated serversforhistorical computing
• Localnodes forreal-timeforecast