IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 43, NO. 6, NOVEMBER 2013
509
Using Temporal Logic and Model Checking
in Automated Recognition of Human Activities
for Ambient-Assisted Living
Tommaso Magherini, Alessandro Fantechi, Chris D. Nugent, Member, IEEE, and Enrico Vicario, Member, IEEE
Abstract—Automated monitoring and the recognition of activities of daily living (ADLs) is a key challenge in ambient-assisted
living (AAL) for the assistance of the elderly. Within this context,
a formal approach may provide a means to fill the gap between the
low-level observations acquired by sensing devices and the highlevel concepts that are required for the recognition of human activities. We describe a system named ARA (Automated Recognizer of
ADLs) that exploits propositional temporal logic and model checking to support automated real-time recognition of ADLs within a
smart environment. The logic is shown to be expressive enough
for the specification of realistic patterns of ADLs in terms of basic
actions detected by a sensorized environment. The online model
checking engine is shown to be capable of processing a stream
of detected actions in real time. The effectiveness and viability of
the approach are evaluated within the context of a smart kitchen,
where different types of ADLs are repeatedly performed.
Index Terms—Activities of Daily Living (ADL), activity recognition, Ambient Assisted Living (AAL), model checking, real-time
processing, temporal logic.
I. INTRODUCTION
HE demographics of the global population are changing,
with an increasing number of both the elderly and people suffering from long-term chronic conditions [32]. Independence, well-being, and health status of these people largely depend on their autonomy in the accomplishment of basic actions
of daily self-care. This level of self care is often evaluated with
reference to the execution capabilities of standard Activities of
Daily Living (ADLs) [24], which address self-care procedures
such as feeding, transferring, and washing [38].
Automated activity recognition [2], [40] applied to the specific scenarios of ADLs has the potential to play a pivotal role
in supporting the quality of life and improving living experience. To this end, a growing number of pervasive technologies,
T
Manuscript received July 2, 2012; revised December 8, 2012 and May 6,
2013; accepted August 17, 2013. Date of current version November 26, 2013.
This paper was recommended by Associate Editor E. Bass of the former IEEE
Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans
(2012 Impact Factor: 2.183).
T. Magherini, A. Fantechi, and E. Vicario are with the Department
of Information Engineering, University of Florence, 50139 Firenze, Italy
(e-mail: [email protected]; [email protected]; enrico.
[email protected]).
C. D. Nugent is with the Faculty of Computing and Engineering, Computer
Science Research Institute and School of Computing and Mathematics, University of Ulster, Belfast, BT37 0QB, U.K. (e-mail: [email protected]).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TSMC.2013.2283661
such as wireless sensor networks (WSN) [19], wired sensor
networks [46], radio-frequency identification (RFID) [34], and
camera-based systems [7], [30], can be used to extract and gather
information from sensorized environments. Nevertheless, a major semantic gap separates the data provided by sensing devices
and the high-level interpretation required for care service delivery. This gap involves different responsibilities and grains of
abstraction, for which various layered architectures have been
proposed [11], [13], [29], [43]. In [11], three levels of concern
are identified, so as to distinguish the acquisition of numerical
observation, the recognition of symbolic observable sequences
corresponding to predefined patterns, and their exploitation in a
higher contextual information level.
Different classifications have been proposed for high-level
pattern-recognition techniques. In particular, two orthogonal categorizations are proposed in [45] and [37]. The former emphasizes the difference between learning-based and
specification-based approaches, while the latter distinguishes
among statistical, syntactic, and description-based approaches.
According to [37], statistical approaches, exploiting models
such as Bayesian Networks [34], Hidden Markov Models [16],
[46], and Conditional Random Fields [41], achieve good results
with sequential activities; however, they require large learning
datasets and present the restrictions of temporal constraint management. Syntactic approaches represent human activities as the
string of symbols and use parsing algorithms for the recognition
of these strings in the information stream. Context free grammar and stochastic context free grammar have been adopted
to recognize activities in [36] and [20], respectively. As main
drawbacks, these approaches need a complete definition of all
possible production rules for each activity, and do not easily
encompass concurrent activities that may occur in several and
different orders. Description-based approaches specify human
activities through properties that shall be satisfied by different realizations. Ontological rules and propositional logic have
been adopted to represent and recognize human activities in [8]
and [35], respectively. Basic guidelines and preliminary results
on the application of propositional temporal logic (TL) and
model checking within the realms of AAL have been proposed
in [28]. As a common trait, activities are characterized through
a higher level of abstraction capturing only a set of salient basic
actions and their spatiotemporal relationships.
In this paper, we propose a novel description-based approach,
where human behavioral patterns are characterized through formulae of propositional real-time TL and checked against the
flow of events detected within a smart environment. This is
2168-2291 © 2013 IEEE
510
IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 43, NO. 6, NOVEMBER 2013
concretely implemented in a prototype system, called ARA
(Automated Recognizer of ADLs), which was used to experiment and test the approach with respect to expressivity and
efficiency of requirements. On the one hand, we show how the
logic is expressive enough to specify realistic patterns of ADLs
in terms of basic actions that can be automatically detected by
a sensorized environment. On the other hand, we also show that
the online model checking engine of ARA is able to manage
observation streams within an experimental setting using stereo
cameras for the identification of the actions of a single person.
With respect to [28], this paper adds: a comprehensive definition of syntax and semantics of the proposed temporal logic, the
development of a system for the automated and real-time recognition of human activities, and an evaluation of the performances
of the proposed approach carried out through a nontrivial dataset
gathered within a smart environment.
The rest of the paper is organized as follows. Section II describes the proposed activity recognition approach based on
a propositional TL and a model checking engine. Section III
shows experimental results concerning the recognition of human
activities in a sensorized environment, and Section IV presents
a discussion that is related to the proposed approach and future
developments that this enables.
the stoves, the oven and the hot-water tap. In this case, the
monitoring system detects that the oven is switched on for more
than 2 h and informs Alice of this situation. She does not switch
off the oven and as a result the SH sends a message to Bob,
informing him about the situation. Finally, Alice switches off
the oven, and the SH sends a new message to Bob informing
him that the potentially dangerous situation is resolved.
While preparing the dinner, the doctor calls Alice; however,
she does not hear the telephone ringing in the living room. The
system recognizes that somebody is in the house and nobody
answers the telephone, and informs Alice that her doctor tried
to call her; however, she did not answer the phone.
Following dinner, the SH reminds Alice to take her prescribed
medication. It subsequently monitors Alice’s actions in order to
verify that she adheres to the reminder.
When Alice goes to sleep, the system checks that the windows
and the front door are closed, and the gas and the lights are
switched off. Finally, the SH saves a log file containing all the
activities performed by Alice in order to build a history of her
behaviors.
II. AUTOMATED RECOGNIZER OF ACTIVITIES
OF DAILY LIVINGS SYSTEM
In the life science context, an ADL is usually considered as
an abstract class of activities with some trait of logical cohesion.
For instance, in [24], the ADL “Transferring” collects together
any concrete activity wherein the subject moves across different
places, which can concretely consist in moving in/out of bed
or going up/down the stairs. In a similar manner, taking clothes
from a wardrobe and putting on overcoats are different instances
of the ADL “Dressing.” The development of automated human
activity recognition techniques requires a more concrete and
detailed specification of ADLs. To this end, we distinguish three
concepts at different levels: activity, action, and observation.
1) An activity is a task undertaken by the observed subject
with an intended goal that may be achieved through a
sequence of different steps, in different orders, and with
different timings. For instance, the activity of brewing an
American coffee normally includes the steps of pouring
water into the cup and adding ground coffee, and possibly
includes the steps of adding sugar and/or milk. The steps of
water pouring and adding the coffee may occur in different
orders, however, drinking will necessarily follow both of
them.
2) An action is a single step possibly characterized by quantitative timing constraints that may occur in different activities. For instance, the action representing water pouring
into a cup is included in the preparation of a hot drink;
however, it is also associated with the activities concerning the intake of medication. Regarding a specific activity,
the actions may be classified into two categories: relevant
and noisy. The former represents the set of actions that the
user may execute during the activity performance, while
the latter includes the actions that do not belong to the
given activity. In particular, a relevant action may be classified as mandatory, if it represents a necessary step for
We introduce functional requirements for AAL through the
description of a hypothetical scenario of application. Then, we
formalize the ADL recognition problem introducing an ontology
for human daily activities which is followed by a definition of
a propositional TL that satisfies the expressive needs of AAL.
We finally report on the implementation of the ARA system
focusing on users and software components.
A. Application Scenario
Alice is an elderly woman, who lives alone. She is selfsufficient, however, her son Bob and her doctor wish to supervise her needs. In particular, they would like to monitor
and support Alice during the performance of daily activities.
Based on this aspiration, her house was transformed to a Smart
Home (SH) through the installation of several kinds of sensing
technologies.
Every afternoon, Alice goes for a walk in a park. After the
walk, Alice returns to her home to prepare a coffee (she does
not like tea). The SH detects that the front door is still opened
and warns Alice, who subsequently closes the door. During the
process of making the coffee, the SH supervizes the activity by
recognizing each of the step performed by Alice and offering
prompts during the instances of confusion or forgetfulness. She
prefers white coffee and adds some cold milk, placed in the
fridge, into the cup. The SH recognizes that the fridge door was
opened and subsequently closed within a short period of time
and infers that an activity was correctly performed by Alice.
After some rest, Alice starts to prepare a roasted chicken
for her evening meal. The SH monitors the users actions in
order to detect the potentially dangerous situations controlling
B. Formulation for the Activities of Daily
Living Recognition Problem
MAGHERINI et al.: USING TEMPORAL LOGIC AND MODEL CHECKING IN AUTOMATED RECOGNITION OF HUMAN ACTIVITIES
the correct execution of the activity, and optional, if it represents a step that may or may not be executed during the
activity performing. Following the aforementioned example, the activity of coffee preparation is composed of two
mandatory actions (water pouring and coffee adding), a
number of optional actions (sugar/milk adding), and a set
of noisy actions (the actions detected by the ARA outside
the kitchen).
3) An observation is a measurement taken by some device
that may occur during the execution of different steps.
For example, a movement of a cup may occur during
the step of water pouring, during drinking, or when the
cup is washed. In general, a single action may result in
multiple observations, possibly stemming from multiple
devices. For example, when a coffee box is taken from
the cupboard, different devices may reveal multiple measurements relating to the acceleration of movement of the
coffee container and on the opening and closing of cupboard doors.
Within this framework, ADL recognition can be formulated as
the problem of identifying whether a sequence of observations,
arising from a sensorized environment, can be interpreted as
the footprint of a set of actions and activities carried out by the
same person. This is largely concerned with the observations,
and also with their sequencing and timing.
C. Temporal Logic for Activities of Daily Living Specification
In order to support an automated approach to the recognition
of ADLs, we have defined a variant of TL as a language for
the specification of the conditions under which a sequence of
observations can be regarded as an occurrence of some action
or activity.
In general, TL has been developed and applied as a formalism
for reasoning on the ordering and quantitative timing of events
[23]. Several formulations have been proposed to satisfy the
needs of different contexts, such as railway signaling systems
[5], avionic systems [31], and software components [22]. TL
may be classified on the basis of the underlying nature of time:
linear temporal logic (LTL) and computational tree logic (CTL).
The former represents the time course as linear (i.e., each time
instant may have only one successor), while the latter represents
the time course as a branching structure, introducing multiple
successors for each present instants.
A bounded verification of a past LTL is shown in [9], where
the introduction of past operators permits a more compact
and simpler specification of formulae, and a bounded history allows the analysis of a fixed number of past time instant. LTL with real-time constraints is presented in [3] and
[26], providing minimum and maximum bounds for temporal
operators.
To satisfy the expressive needs for ADL specification, we
extended and combined bounded history, past operators, and
real-time constraint features, proposed in [3], [9], and [26],
respectively, to define syntax and semantics of TL based on the
following characteristics.
511
1) Propositional TL: we will reason about actions in the
observed environment, combined through Boolean and
temporal operators. First order predicates may also fit the
context (e.g., in the management of classes of equivalent
objects or actions); however, at the expense of a much
higher complexity.
2) Past oriented: as opposed to most formulations, our logic
shall reason about the past rather than future, using the past
operators: Y (Yesterday), O (Once), H (Historically), and
S (Since). The use of these operators does not change the
expressive power with respect to a conventional futureoriented logic; however, it enables a more convenient formulation of ADLs.
3) Linear time: we focus on the flow of actions executed by
the users and the environment, which is a linear sequence
by construction.
4) Discrete time: actions are detected by sensing devices in
an asynchronous manner at time instants which consider
values in a continuous domain. Nevertheless, each action
can be safely discretized with a cost effective tradeoff
between efficiency and accuracy, producing subsets of actions and activities characterized by the same time instant.
5) Real-time constraints: in order to capture quantitative constraints of the time elapsed between actions, the logic shall
include real-time operators. In particular, we assume temporal operators subscripted with time annotations as usual
in real-time TL [3].
6) Bounded history: we focus on the actions executed by
the user in the recent past, given that expected patterns of
ADLs are supposed to be encompassed within a bounded
time span. For this purpose, the logic shall consider only
a finite part of the action flow.
The resulting logic has been referenced as bounded Real-time
past linear Temporal Logic (referred to as R-pTL).
The syntax of the R-pTL is defined over a set of atomic
proposition AP = {p, q, . . .} and a set of timed instants T =
{t1 , t2 , . . .} ⊆ N
Definition 2.1: Let Σ be a set of timed states. A timed state
σt ∈ Σ is a pair (s, t), where s ∈ 2A P is the set of atomic
propositions that are true at time t ∈ T [26].
Definition 2.2: A timed trace Γ ⊆ Σ, is a possible infinite
sequence of timed states
Γ = . . . , σi−1 , σi , σi+1 , . . .
where σi ∈ Σ, and σi = (si , ti ) ∀i : ti ≤ ti+1 .
Definition 2.3: A Bounded Timed Trace (BTT) ΓB ⊆ Σ over
AP, is a finite sequence of Timed States:
ΓB = σt−B +1 , σt−B +2 , . . . , σt
where B ∈ N is the number of timed states in ΓB , and t is the
current timed instant. We assume that t ≥ B, which means that
the current time instant t is at least B time units after the start
of the time basis (t = 0).
R-pTL formulae are formed by combining atomic propositions through Boolean connectives (∧, ∨, and ¬), past temporal operators Yesterday (Y), Once (O), Historically (H), and
Since (S), possibly extended with real-time constraints (O[c,d] ,
512
IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 43, NO. 6, NOVEMBER 2013
TABLE I
R-pTL SEMANTICS: A TIMED STATE σ t IN A KRIPKE STRUCTURE M SATISFIES (|=) DIFFERENT FORMS OF R-pTL FORMULAE φ (LEFT COLUMN)
IFF σ t BELONGS TO SET Σ B OF THE TIMED STATES VISITED IN THE PAST WITHIN THE TIMED BOUNDARY B, AND SATISFIES
SOME OTHER CONDITIONS DEPENDING ON THE SPECIFIC FORM OF φ (RIGHT COLUMN)
H[c,d] , and S[c,d] ). The meaning of each formula is presented in
Table I.
Definition 2.4: (R-pTL Syntax) An R-pTL formula φ is inductively defined as follows:
φ ::= p |¬φ| φ1 ∧ φ2 |φ 1 ∨ φ2
φ ::= Yφ |Oφ| Hφ |φ1 Sφ2
φ ::= O[c,d] φ H[c,d] φ φ1 S[c,d] φ2 where p ∈ AP and c, d ∈ Z− , with c ≤ d.
The semantics of the R-pTL formulae is defined with respect
to a Kripke structure M = (ΣB , RB , L) that characterizes the
behavior of a system in terms of a finite set of timed states
ΣB ⊆ Σ; a transition relation RB = {(σx , σy )|σx , σy ∈ ΣB and
x = t + i − 1, y = t + i with −B + 1 ≤ i ≤ 0}; and a labeling
function L : ΣB → 2A P given that L(σi ) = s iff σi = (s, t), that
indicates which atomic propositions are satisfied in each timed
state. We also define a function T : Σ → N that associates each
timed state σi with its time t (T (σi ) = t iff σi = (s, t)).
Let φ be an R-pTL formula and σt a timed state. The semantics
of the statement “M accepts the formula φ in the Timed State
σt ,” denoted by M, σt |= φ, is defined by structural induction
on φ as shown in Table I.
The usual relationships among Boolean operators and
among temporal operators are satisfied. For example H[c,d] φ =
¬(O[c,d] ¬φ) and O[c,d] φ = True S[c,d] φ. All operators of the
logic can be expressed with a combination of operators belonging to the following minimal set: {¬, ∧, Y, S[c,d] , S}. A graphical representation of examples of the past temporal operators
with real-time constraints is shown in Fig. 1.
MAGHERINI et al.: USING TEMPORAL LOGIC AND MODEL CHECKING IN AUTOMATED RECOGNITION OF HUMAN ACTIVITIES
(a)
(b)
(c)
(d)
Fig. 1. Graphical representation of the past temporal operators. The Kripke
structure is graphically represented by linear paths, in which the nodes σ t
(shown in black) represent the present timed states, while the nodes σ i (shown
in white) with i = t represent the past timed states. States are also subscripted
by the formulae φ i that they satisfy. (a) the R-pTL formula Yφ is verified by
the model given that in σ t −1 , the previous timed state of σ t , φ is true; (b) the
R-pTL formula O [c , d ] φ is verified by the model given that for at least a timed
state σ j between σ t + c and σ t + d , φ is true; (c) the R-pTL formula H [c , d ] φ
is verified by the model given that for each timed state σ j between σ t + c and
σ t + d , φ is true; (d) the R-pTL formula φ 1 S [c , d ] φ 2 is verified by the model
given that for at least a timed state σ j between σ t + c and σ t + d , φ 2 is true, and
φ 1 is true for each timed state from σ j + 1 to σ t + d .
D. Online Model Checking System for Real-Time
Past Linear Temporal Logic Specification
We developed a context independent system, referred to as
ARA, which is designed to verify the properties of the model in
an online manner in addition to raising alerts when an abnormal
situation is detected. The ARA implements a model checking
engine for the verification of R-pTL formulae with the aim of
supervising the activities performed by users on the basis of the
information extracted by a sensorized environment.
The recognition of potentially dangerous activities in an online fashion allows timely delivery of appropriate interventions.
For example, referring to the application scenario presented in
Section II-A, when Alice forgets to take the prescribed medication, the system informs her relatives of her lack of adherence
to the medication regime.
The ARA represents the user’s activities using R-pTL formulae and the information coming from the sensorized environment
is represented through a labeled directed graph. In particular, the
model is represented by a BTT, which is periodically updated
by adding a new timed state containing the recognized actions
in the previous time interval and by deleting the oldest timed
state. Following this, our system applies the model checking algorithm for each R-pTL formula ensuring that the BTT is kept
up to date.
Based on the guidelines presented in [6], we classify the users
interacting with our system in four classes: Caregiver, Analyst,
Environment, and clock (see Fig. 2).
1) The Caregiver is the main involved user, supervising daily
activities performed by assisted users through the ARA
system. He/she is an expert in the care domain and has
a sufficient knowledge of the system features, however,
may also have a limited knowledge from a technological
perspective.
513
2) The Analyst examines action patterns performed by assisted users and context, and formalizes them as TL statements. He/she has a good knowledge of the ARA system
and a sufficient knowledge of the care domain; however,
he/she may not fully appreciate the assisted user and specific technological details.
3) The Environment interacts with the ARA system by sending actions extracted by the sensing technologies within
the Environment and receiving feedback created by the
system or the Caregiver.
4) The Clock interacts with the ARA system by requesting
the model update and the formula verification on the basis
of an updating interval.
The component diagram of Fig. 3 illustrates the three main
components of the ARA system: Formula Editor, Model Manager, and Model Checker.
The Formula Editor helps the Analyst to represent ADLs
using R-pTL formulae. In particular, the formula GUI component provides a graphical user interface which aids the Analyst to write and modify R-pTL formulae. The Syntax Assisted
Editor component verifies syntactic correctness of the formulae, detecting errors, and proposing alternative correct versions.
For example, if the Analyst writes the formula φ = aHb, the
Syntax Assisted Editor will recognize that the formula does
not respect the R-pTL syntax and will suggest the following
R-pTL formula φ = Hb. The Context Analyzer component verifies the consistency of the formulae on the basis of the timed
features of the model. For instance, if the Analyst uses the
real-time version of the temporal operators introducing realtime constraints in the order of minutes when the model is
updated every hour, then the Context Analyzer will alert the
the Analyst to the dimensional inconsistency of the real-time
parameters.
The Model Manager creates, updates, and modifies the model,
using the set of actions extracted from the environment by the
Action Manager component. In particular, the Action Manager
provides the middleware between the Observation Gateway and
the Timed State Generator components, making the observations gathered from the environment independent with respect
to the model needed for the recognition process. Context independence permits the system to be robust with respect to
changes in the sensorized environment (e.g., the replacement
of a wireless sensor network with a camera-based system does
not require manual configuration of the system), and enables
system portability on various employment contexts (e.g., a flat
instead of an isolated house). The concept of using middleware
to achieve context independence is based on [44]. The extracted
actions are represented by atomic propositions as prescribed
by the R-pTL syntax and collected into a new timed state created by the Timed State Generator component on the basis of
updating the time interval assigned by the Analyst. The model
represented by a BTT is managed by the Model Online Updater
component, which adds the new timed state provided by the
Timed State Generator and deletes the oldest timed state of the
model in order to consider only the recent past. In particular,
the bounded history of the R-pTL logic is managed in the ARA
through the maximum number of timed states belonging to the
514
IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 43, NO. 6, NOVEMBER 2013
Fig. 2. Use Case diagram of the ARA system showing four classes of users: Caregiver, Analyst, Environment, and clock. In particular: (a) the Caregiver monitors
the assisted user consulting results log and feedback provided by the ARA system; (b) the Analyst writes, modifies, and deletes R-pTL formulae, and configures
the model by setting the updating time interval and the span of the bounded history; (c) the Environment sends observations gathered from sensing devices to the
ARA system; (d) the clock updates the model and verifies the R-pTL formulae on the basis of an updating time given by the Analyst.
Fig. 3. Component diagram of the ARA system showing the three main components a Formula Editor helps Analyst to represent ADLs through R-pTL formulae,
a Model Manager creates, updates, and modifies the model, and a Model Checker verifies whether the model satisfies the R-pTL formulae.
model, which must contain the activity with the maximum user
performance time.
The Model Manager and the formula editor provide the model
(i.e., the model component) and the well-formed set of R-pTL
formulae (i.e., formula set component) to the Model Checker,
respectively.
The Model Checker is the core of the system and recognizes
the activities performed by the assisted user. In particular, the
Model Checking Engine component verifies whether the model
satisfies the R-pTL formulae through an extended version of
the algorithm presented in [10]. To handle an arbitrary R-pTL
formula φ, the Model Checking Engine component iteratively
splits φ into subformulae, assigning them a length value determined by counting the number of the operators that compose
the formula. A formula, composed by a single atomic proposition, has length equal to 0, while a formula that includes two
operators, e.g., φ = X(p ∧ q) where p, q ∈ AP, has length equal
to 2. Starting with the shortest formula and applying an iterative
approach, the algorithm assigns each subformula to the timed
states wherein it is true, ensuring when φ is processed, all its
subformulae have already been assigned. The final result of the
algorithm is the set of timed states wherein the formula φ is
satisfied. Each iteration takes time O (B + |RB |), where B is
the number of the Timed States σt composing the BTT ΓB , and
|RB | is the number of transitions. In our case, |RB | = B − 1
gives that the model is represented by a bounded linear path. The
complexity of each iteration becomes O(B + B − 1) = O(2 ∗
B − 1) = O(B). Hence, if |φ| is the number of subformulae of
MAGHERINI et al.: USING TEMPORAL LOGIC AND MODEL CHECKING IN AUTOMATED RECOGNITION OF HUMAN ACTIVITIES
φ, then the algorithm requires time O(B ∗ |φ|) for processing
the formula φ.
Finally, after the formulae verification process the Feedback
and Result Publisher component creates a result log of the recognized activities used by the Caregiver to evaluate the health status of the assisted user, and provides a web interface to interact
with the assisted user and smart environment. This component
periodically sends the list of missed activities to the assisted user
through several communication devices, such as smartphones,
mobile devices, and house monitors. The proposed system may
be further extended for reminder support [33]. Using the web
interface, the Caregiver may interact with the sending devices
integrated into the environment. For instance, he/she may switch
on cameras and interrogate a sensor within a WSN in order to
manually detect activities performed by the assisted user.
III. EXPERIMENTAL ASSESSMENT
Effectiveness and viability of the proposed approach have
been experimentally evaluated with the dual aim of verifying
of the capability of R-pTL in expressing realistic patterns and
the ability of the Model Checking Engine to operate in an online mode. To this end, six activities, extracted from the scenario
proposed in Section II-A, are represented using R-pTL formulae
and tested within a sensorized environment. The selected activities are complex and sufficiently diverse to exercise and stress
the expressivity of different aspects of the proposed R-pTL.
Experiments were carried out to evaluate the Recall and Precision of the automated recognition engine and to analyze its
ability to operate in an online fashion.
A. Representation of Activities of Daily Livings Into
Real-Time Past Linear Temporal Logic Formulae
The specification of ADL abstract classes, proposed in [24],
is primarily intended to comprise a classification framework and
thus avoids details on the way how activities occur. In order to
obtain a solid ground for automated recognition of activities, we
thus refer to [34] and [42], which provide a range of notable and
detailed ADL lists. We focused on the following three ADLs:
1) Coffee preparation concerns the activities of American
coffee preparation, which may be deemed as being correct
or incorrect depending on the ordering and the timing of
the actions.
2) Telephone usage concerns the activities of answering the
telephone, distinguishing the case of correctly and incorrectly (Missed call) phone answering. Both activities
are characterized by an ordered and timed sequence of
actions.
3) Take medication concerns the intake medication, hence the
automation of the process distinguishes the case of correctly and incorrectly taking medication. Both activities
involve time and order constraints of actions.
The modeling process regarding the translation from natural language specification of activities [34], [42] to R-pTL
formulae can be addressed through the following three-phase
methodology.
515
Given a natural language representation of a human activity,
the first step is the identification of actions that compose the
activity and their classification as being either mandatory or
optional. The actions that do not belong to the given activity
are labeled as noisy. The second step regards the enumeration
of admissible sequences of mandatory actions, which can be
categorized as ordered (e.g., the user must execute the actions
a, b, and c respecting a given pattern), not-ordered (e.g., the
user may execute the actions a, b, and c with an arbitrary
order), or semiordered (e.g., the user must execute the actions a
and then b, while c may be executed whenever the user likes).
Each admissible sequence may be translated into a specific RpTL pattern (e.g., the logic pattern c ∧ (Ob ∧ (Oa)) may
represent the ordered action sequences a, b, and c). The third
step introduces quantitative time constraints using the real-time
version of temporal operators, which are able to comprise the
maximum execution time of the given activity and the timed
intervals for the action execution into R-pTL formulae.
1) Coffee Preparation: We illustrate the modeling process
with reference to the activity of correct preparation of a coffee.
We initially identify the mandatory actions, i.e., those actions
that must necessarily occur during the execution of a given
activity. In this case, the water pouring into the cup (cpw) and the
addition of the ground coffee (cpc) are deemed to be mandatory.
We then focus on the admissible sequences of the mandatory
actions. In this case, cpw and cpc can occur in any order
and these possible sequences can be captured by the following
R-pTL formula:
(Ocpw) ∧ (Ocpc)
(1)
which is true if both cpw and cpc have occurred in any order at
least one time in the past (operator Once, denoted by O). Finally,
we add quantitative time constraints assuming a maximum time
tce within which the user is supposed to prepare the coffee. This
extends the specification with the real-time version of the past
temporal operators:
(O [−tce , 0] cpw) ∧ (O [−tce , 0] cpc) .
(2)
In a similar manner, the Incorrect coffee preparation activity is
captured by the following R-pTL formula:
((O[−tce , 0]cpw) ∧ (¬(O[−tce , 0]cpc)))
∨
((O[−tce , 0]cpc) ∧ (¬(O[−tce , 0]cpw)))
(3)
where the former disjunctive clause represents the case in which
the user pours water into the cup, however, forgets to add coffee,
while the latter disjunctive clause represents the case wherein
the user adds coffee, however, forgets to pour in water.
According to (3), the specification of the Incorrect coffee
preparation is satisfied by any intermediate states where the
user is proceeding correctly, however, the activity has not been
completed. This drawback can be formalized as follows:
T (σi ) < T (σj ), cpw ∈ L(σi ), cpc ∈ L(σj )
σ i , σj ∈
B,
516
IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 43, NO. 6, NOVEMBER 2013
(a)
(a)
(b)
(b)
Fig. 4. Graphical representation of two instances of the Coffee preparation
ADL: (a) the model represents the instance wherein the assisted user pours water
into the cup (cpw), and subsequently he/she adds coffee (cpc). In this case,
the R-pTL formula (2) representing the Correct coffee preparation activity is
satisfied by the model; (b) the model represents the instance wherein the assisted
user adds coffee to the cup, however, he/she forgets to pour water. In this case,
the R-pTL formula (4) representing the Incorrect coffee preparation activity is
satisfied by the model.
Fig. 5. Graphical representation of two instances of the Telephone usage ADL:
(a) the model represents the instance wherein the telephone rings (pri) and the
assisted user answers it (pan). In this case, the R-pTL formula representing the
Telephone answered activity is satisfied by the model; (b) the model represents
the instance wherein the telephone rings for at least a time tp r, however, the
assisted user does not answer it. In this case, the R-pTL formula representing
the Missed call activity is satisfied by the model.
and φ represents the R-pTL formula (3), then ∀ σh ∈ ΣB with
T (σi ) ≤ T (σh ) < T (σj ) M , σh |= φ, which is a detection
error. To overcome the limitation, we will evaluate the actions
executed by the user only when a sufficient delay time has
elapsed. To this end, the formula (3) is rewritten as:
where action pri must be satisfied for all the timed states
belonging to time interval [0 − tpa , 0 − tpa + tpr ], and it returns
true if and only if the telephone rang for at least a time tpr . While
the second sentence is represented as:
((H [−tcd , −tcd ] cpc) ∧ (¬ (O [−tce , 0] cpw)))
∨
((H [−tcd , −tcd ] cpw) ∧ (¬ (O [−tce , 0] cpc)))
(4)
where tcd is the delay time and tcd ≤ tce . The formula is composed of two disjunctive clauses. The former represents the case
in which the user pours water into the cup at time t − tcd , however, forgets to add coffee during the time interval [t − tce , t],
while the latter represents the case where the user adds coffee at
time t−tcd , however, forgets to pour water during the aforementioned time interval. In particular, H[−tcd , −tcd ]cpc is true if
and only if the user pours water into the cup at the timed state
characterized by the time instant t − tcd .
Fig. 4 shows two models representing the Coffee preparation
ADL.
2) Telephone Usage: The target of this ADL is the recognition of two activities: the telephone rings and the user answers
(Telephone answered); and, the telephone rings and the user
does not answer (Missed call). Both are characterized by the
following mandatory actions: the phone is ringing (pri), and
the user answers the telephone (pan).
The Missed call activity accepts a unique possible sequence
of actions: the telephone was ringing and the user did not answer
it. We assume that tpr is the minimum time duration wherein
the telephone must ring to be considered a telephone call and
tpa is the maximum time duration wherein the user has to answer the telephone. The activity is then formalized as follows:
the telephone rang for at least a time tpr and the user has not
answered it in a time tpa . The activity can be subsequently split
in two simpler sentences:
1) the telephone rang for at least a time tpr ;
2) the user has not answered the telephone in a time tpa .
The first sentence is represented by the following R-pTL
formula:
H[−tpa , −tpa + tpr ]pri
(5)
¬O[−tpa + tpr , 0]pan
(6)
where O[−tpa + tpr , 0]pan is true if and only if the user has
answered in the interval [−tpa + tpr , 0] and the ¬ operator
has been used to recognize when the user has not answered the
telephone. Finally, the activity is represented by the conjunction
of the previous formulae:
(H[−tpa , −tpa + tpr ]pri) ∧ (¬(O[−tpa + tpr , 0]pan)).
(7)
A different approach has been considered for the recognition
of the activity of the telephone being answered. This activity is
characterized by the following sentence: the telephone is ringing
and the user answers it. The formula is based on the following
assumption: if the user answers the telephone, it suggests that
the telephone was previously ringing. In a first formulation this
may be captured as
(Ypri) ∧ pan
(8)
which is true if pan occurs in the present timed state and pri
has occurred in the previous timed state (operator Yesterday,
denoted by Y). Nevertheless, in doing so, the activity is not
correctly recognized when the user executes both actions in the
same time or in a short time interval. This is due to the temporal
granularity of the past. A timed state includes the actions executed by the user in a previously prefixed time interval; hence, a
timed state could contain both pri and pan actions. To capture
this case, a subformula (pri ∧ pan) is added, which is true if
and only if pri and pan are true at the same timed state. The
formula (8) was rewritten as
((Ypri) ∧ pan)) ∨ (pri ∧ pan).
(9)
Fig. 5 shows two models representing the Telephone usage ADL.
3) Take Medication: This scenario is focused on medication
compliance. We consider two activities: the user correctly takes
his/her prescribed medication and the user does not take his/her
prescribed medication. Both activities can be described by a
combination of the following three actions: the user takes a pill
(mtp), the user pours water into a cup (mpw), and the user drinks
MAGHERINI et al.: USING TEMPORAL LOGIC AND MODEL CHECKING IN AUTOMATED RECOGNITION OF HUMAN ACTIVITIES
517
(a)
(b)
Fig. 7.
Fig. 6. Graphical representation of two instances of the Take medication ADL:
(a) the model represents the instance wherein the assisted user pours water into
a cup (mpw), subsequently he/she takes a pill from the dispenser (mtp), and
finally he/she drinks water from the cup (mdw). In this case, the R-pTL formula
representing the Correct medication intake activity is satisfied by the model;
(b) the model represents the instance wherein the assisted user pours water and
takes a pill, however, he/she forgets to drink some water. In this case, the R-pTL
formula representing the Incorrect medication intake activity is satisfied by the
model.
water (mdw). We assume that tm e is the maximum time within
which the user must complete these activities.
The Correct medication intake activity is formalized as follows: the assisted user drinks water and in the previous time
interval [−tm e , 0] he/she has taken a pill from the drug dispenser, and he/she has poured water into the cup. This activity
can be represented by the following formula:
mdw ∧ (O[−tm e , 0]mpw) ∧ (O[−tm e , 0]mtp).
(10)
The R-pTL formula describing the Incorrect medication intake
activity is based on the approach previously described for the
Incorrect coffee preparation. As such, the activity is represented
by the following action sequence: the user took medication from
the medication dispenser, however, he/she has forgotten to pour
water from a bottle to a cup or he/she forgets to drink some
water from the cup. The activity is represented by the following
formula:
(H[tm d , tm d ]mtp)
∧
(11)
((¬(O[−tm e , 0]mpw)) ∨ (¬(O[tm d , 0]mdw)))
where tm d is the delay time for this activity.
Fig. 6 shows two models representing the take medication
ADL.
B. Experimental Results in a Smart Environment
We tested and evaluated the ARA system on a dataset created using the facilities available within the Smart Environments
Research Group (SERG) at the University of Ulster [32]. The
environment is equipped with a range of embedded sensor-based
and actuator technologies, display panels, video- based recording, and tracking systems. For the purposes of experimentation
a video-based tracking system, referred to as DANTE [12] was
used to evaluate the proposed approach.
1) DANTE Activity Monitoring System: DANTE is a software application [12] which supports the monitoring and annotation of human activities within a smart environment captured
via a pair of stereo cameras [15]. While support for a single camera exists, the incorporation of two cameras reduces instances of
object occlusions and supports an increased field-of-view within
Example of DANTE markers used to tag objects.
the environment being monitored. In our configuration, the field
of measurement detects markers within a range of 0.5 and 5 m,
and can be configured to sample at a frame rate of up to 20 Hz.
Up to 100 objects can be concurrently tracked by the system;
however, this limitation is attributed to the processing power of
the system running DANTE, not the DANTE system itself.
Using a unique reference point within the environment, the
two cameras can interpret both the location and orientation of
objects in a coordinate space. Objects are “tagged” with paperbased markers, which consist of unique black and white pattern
coding (see Fig. 7). Each marker is learnt by the camera system
while in an offline mode and labeled to identify a specific object,
to which it is attached. Objects in the current context refer to
any item within a home environment, which may be viewed as
being related to an ADL.
Following recording, a scenario within the environment, an
operator can use DANTE to navigate frame-by-frame through
captured scenes to annotate the onset/completion of object interactions. The main utility of this feature of the system is to
support the development of large annotated datasets, which is
deemed to be essential for the development and evaluation of
context-aware models. In our study, we have used DANTE to
construct an annotated dataset for the purposes of experimental
evaluations.
2) Dataset Creation: The dataset was generated using the
facilities within the kitchen environment of the SERG Lab. Several objects were tagged with DANTE markers in the kitchen:
glasses, cups, coffee/sugar/milk containers, bottles, pill dispensers, fridge door, and cupboards. In total 20 objects were
tagged. The experiments focused on six activities: Correct coffee preparation, Incorrect coffee preparation, Telephone answered, Missed call, Correct medication intake, and Incorrect
medication intake.
Three subjects participated in the evaluations. Each participant was informed about a set of mandatory actions that they
should carry out in addition to leaving them free to decide individually about nonmandatory actions which they may wish
to undertake and about the possible sequence with which the
activities may be executed. Each subject performed a randomly
selected subset of the activities 12 times. The execution of an
activity includes the execution of the mandatory actions; however, the execution of the optional and noisy actions is totally
dependent upon the preferences of the participants. For example,
an instance of the correctly preparing coffee activity includes
the mandatory actions of water pouring and adding coffee. This
activity may include optional actions, such as adding sugar and
milk, and noisy actions, such as moving bottle or kettle.
518
IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 43, NO. 6, NOVEMBER 2013
(a)
(b)
(c)
(d)
Fig. 8. DANTE Frame videos regarding to the Correct coffee preparation activity. Two cameras were installed in the smart kitchen in order to detect the assisted
user actions. (a) the Operator user has initialized DANTE system, which detects the markers attached to the objects, (b) the Assisted user pours water of the kettle
into the cup. DANTE detects the changes of position and orientation of kettle and cup, (c) the Assisted user adds coffee to the cup. The camera system detects the
position and orientation changes of the coffee box and cup, (d) the Assisted user drinks the coffee. DANTE detects the position and orientation changes of the cup.
TABLE II
OVERVIEW OF THE FORMULAE RELATED TO THE SIX ACTIVITIES ADDRESSED
IN THE EXPERIMENTATION
Fig. 9. This model includes two instances of coffee adding action (cpc) and
one instance of water pouring action (cpw), representing the case in which the
user correctly prepares a coffee, and subsequently he/she adds coffee to the cup
but he/she forgets to pour water.
TABLE III
EXPERIMENTAL RESULTS: EFFECTIVENESS OF RECOGNITION FOR
DIFFERENT ACTIVITIES
The ground truth of the recordings was established by manually annotating the actions performed by the subjects through
the DANTE system (see Fig. 8). In total, for all participants, a
total of 549 actions were collected. The list of annotated actions
was used as input to ARA, which considered these actions as
the atomic propositions of the R-pTL. The ARA generates an
output indicating time instances wherein the model satisfies the
RpTL formulae, which represents the activities.
3) Results: The activity recognition performance of the
ARA system was evaluated through the computation of true positives (TP), false positives (FP) and false negatives (FN), which
occur when the system correctly recognizes an activity as having
performed; recognizes an activity that has not been performed;
and does not recognize an activity that has been performed, respectively. Moreover, we calculate the Precision representing
the fraction of recognized activities that are performed by a user
TP
), and the Recall representing the fraction
(Precision = TP+FP
of the activities which are performed that are recognized by the
TP
).
ARA (Recall = TP+FN
Experimentation results are presented in Table III.
The evaluation phase highlights that the ARA may score an
FP when an action contributes to the patterns of multiple succeeding activities. For instance, if the assisted user correctly
prepares a coffee, and subsequently he/she starts to repeat the
same activity, forgetting to perform a mandatory action (see
Fig. 9), then the ARA subsequently erroneously recognizes two
instances of the Correct coffee preparation activity, although it
should only recognize a Correct and an Incorrect coffee preparation activity. The complexity of R-pTL formulation of the
Incorrect medication intake activity emphasizes this limitation,
prejudicing the recognition performance of the ARA. In principle, these limitations could be overcome with a logic that is
able to identify distinct occurrences of the same type of events
so as to permit the assignment of actions to the respective activity. Nevertheless, to this end the logic should be extended from
the propositional to first-order formulation, thus increasing the
runtime effort of checking.
The ARA scores an FN when an action, which has occurred,
is not detected by the sensorized environment. This is due to the
limitations of the sensing technologies, such as occlusions for
camera-based systems, or transmission errors for WSNs. FNs
may be reduced by improving and combining sensing technologies, for instance, the movements of an object may be detected
by integrating both a video object tracking system and a movement sensor. Regarding the Correct medication intake activity,
our system presents only FN errors, which are due to the lack
of recognition of at least one of the three mandatory actions
characterizing this activity: the user takes a pill (mtp), the user
pours water into a cup (mpw), and the user drinks water (mdw).
Table IV presents performance results, which are based on
the recognition of the formulae presented in Table II, reporting
both average (AvET) and maximum execution time (MaxET)
MAGHERINI et al.: USING TEMPORAL LOGIC AND MODEL CHECKING IN AUTOMATED RECOGNITION OF HUMAN ACTIVITIES
TABLE IV
EXPERIMENTAL RESULTS: PERFORMANCES OF THE ON-LINE MODEL
CHECKING ENGINE
spent by the ARA, with respect to the number of concurrent
R-pTL formulae (# formulae), the number of the subformulae
(# subformulae), and the number of timed states that belong to
the BTT (# timed states). The experiments are performed with
different values of the number of formulae, repeating the six
formulae presented in Table II, and of the timed states belonging to the BTT, in order to stress the computational and memory
resources of the system. The updating time of the BTT is set
equal to 10 s, which is coherent with the time constraints of
the activities examined in Section III-A. Moreover, we chose
the following values for the formulae time parameters: the maximum time tce within which the user is supposed to prepare
the coffee and the maximum time tm e within which the user
is supposed to take the medication pills are set equal to 900 s
and 300 s, respectively; while, the minimum time duration tpr
wherein the telephone must ring to be considered a telephone
call and the maximum time tpa within which the user has to answer the telephone are set equal to 4 s and 30 s, respectively. The
delay time of the coffee preparation activity tcd and medication
intake activity tm d is set equal to half of the maximum times
tce and tm e , respectively. Empirical studies suggest that setting
the delay time of a given activity equal to half of its maximum
execution time reduces mismatch detection errors. In particular,
experiments show how a different choice of these delay values does not significantly change recognition performance of
ARA, with the exception of values close to 0 or to the activity
maximum execution time. The adoption of these edge values
produces detection errors during the ARA formula verification
process, given that intermediate states may satisfy a formula
although the given activity has not been completed.
The experimentation results show how the execution time is
strictly dependent on both the dimension of the BTT and on
the numbers of formulae, which are related to the application
context. In fact, the BTT must contain the activity with the
maximum user performance time, while the number of formulae
is related to the number of human activities to be recognized.
For all the experiments, the recognition process was carried out
in less than a second. Experimentations were performed on an
HP notebook with a dual core T9400 at 2.53 GHz and 4 GB
RAM.
519
IV. DISCUSSION
Automated recognition of human activities and behaviors
within sensorized environments is rapidly being considered as
a core component within the ambient-assisted living paradigm.
Nevertheless, effective deployment of an activity recognition
system still appears limited due to a semantic gap between the
measured data provided by sensing devices and the information
required to deliver care services.
We experimented how formal methods can be adopted to
provide a formal ground to reduce this gap. To this end, we
developed and tested a system referred to as ARA based on an
R-pTL in addition to an online model checking engine. To the
best of our knowledge, propositional TL approaches for pattern
recognition have to date not been presented.
The use of a past TL approach has been found to be a suitable approach in supporting the representation of the ADLs
in the form of logic formulae, given that the ARA recognizes
the activities performed by the user in the past. Furthermore,
the definition of time constraints adds expressive power to the
logic, allowing the introduction of quantitative bounds of the
time elapsed between actions. The representation of activities
through R-pTL formulae, requires a basic knowledge of the
logic. Nevertheless, the complexity involved in the definition of
the activities can be reduced adopting a general methodology to
translate the natural language specification of human behaviors
into R-pTL formulae. In particular, we propose a methodology composed of three steps: identification of mandatory actions, enumeration of possible sequences, and adding of time
constraints. Automated derivation of real-time temporal logic
formulae from pattern-based expressions [1], [17] may largely
smooth the complexity of our approach and improve its usability
for user experienced in ADL more than in formal methods.
The introduction of an online model checking engine produces relevant benefits to the automated supervision of people
requiring medical attention. Such a concept supports the recognition of human activities in a real-time manner and the raising
of an alert when a potentially dangerous situation is detected.
In addition, our system is independent from the adopted sensing technologies and thus can be ported to several application
domains without the requirement of a dataset to train the model.
We tested the ARA on a dataset created in a smart environment
on the basis of the repetitions of six activities extracted from
our referred scenario, which are sufficiently varied to exercise
and stress the expressivity of different aspects of the proposed
logic. With the aim of simulating human realistic behaviors,
three subjects performed 12 times a randomly selected subset of
these activities, executing a range of optional and noisy actions,
in addition to the mandatory actions. The resulting dataset is
composed of 549 actions as detected by DANTE camera system. The ARA correctly recognizes the 85.4% of the activity
instances.
The ARA offers a formal ground to reduce the semantic gap
between raw data and high-level concepts. Nevertheless, automated techniques for activity recognition are still affected by
noisy data gathered from low-level sensing devices and imperfect information obtained by data refinement. The presence of
520
IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 43, NO. 6, NOVEMBER 2013
imperfect information, that may represent vagueness of the characterization of expected human behavior patterns, imprecision
of detected measurements, and uncertainty of possible missed
and misclassified events [25], [39], makes the recognition process more complicated.
We are now working to address the problem of uncertain information and state-space explosion of quantitative models. The
former may be reduced by the adoption of discount computational tree logic [14], [21], which is a multivalued generalization of the branching time logic CTL, or probabilistic temporal
logics, such as probabilistic computational tree logic [18] and
continuous stochastic logic [4], which describe properties over
Markovian models. The latter may be faced by the introduction
of imperfect structures (i.e., sequences of elementary observations stemming from the sensorized environment) and statespace reduction techniques. In particular, we wish to formalize
these problems as a probabilistic model checking process on a
discrete time Markov chain [27] using well-established analysis
techniques that may gain benefits from the bounded observation history and by the fact that we are only interested in events
which overcome a given probability value.
Recognition of human activities and behaviors still present
several challenges which are largely dependent on the underlying complexity of the tasks performed by human users. For
instance, a user may execute two or more activities simultaneously (concurrent activities) or switch between the steps of two
or more activities (interleaved activities) and may perform these
tasks autonomously or collaborating with other users (multiuser
activities). The probabilistic extension of our framework manages imperfect information and increases expressive power of
the formal core, allowing the representation and recognition of
complex human activities.
REFERENCES
[1] N. Abid, S. D. Zilio, and D. L. Botlan, “Real-time specification patterns
and tools,” in Formal Methods for Industrial Critical Systems. New
York, NY, USA: Springer, 2012, pp. 1–15.
[2] J. K. Aggarwal and M. S. Ryoo, “Human activity analysis: A review,”
ACM Comput. Surv., vol. 43, no. 3, pp. 16:1–16:43, Apr. 2011.
[3] R. Alur and T. A. Henzinger, “Logics and models of real time: A survey,”
in Proc. Real-Time: Theory in Practice, REX Workshop, 1992, pp. 74–106.
[4] A. Aziz, K. Sanwal, V. Singhal, and R. Brayton, “Verifying continuous
time Markov chains,” in Computer Aided Verification. New York, NY,
USA: Springer, 1996, pp. 269–276.
[5] C. Bernardeschi, A. Fantechi, S. Gnesi, S. Larosa, G. Mongardi, and
D. Romano, “A formal verification environment for railway signaling
system design,” Formal Methods Syst. Des., vol. 12, pp. 139–161, 1998.
[6] N. Bevan and M. Azuma, “Quality in use: Incorporating human factors
into the software engineering lifecycle,” presented at the 3rd Int. Software
Engineering Standards Symp., Washington, DC, USA, 1997.
[7] J. Boger, P. Poupart, J. Hoey, C. Boutilier, G. Fernie, and A. Mihailidis, “A
decision-theoretic approach to task assistance for persons with dementia,”
in Proc. Int. Joint Conf. Artif. Intell., 2005, pp. 1293–1299.
[8] L. Chen, C. D. Nugent, M. Mulvenna, D. Finlay, X. Hong, and M. Poland,
“Using event calculus for behavior reasoning and assistance in a smart
home,” in Proc. 6th Int. Conf. Smart Homes Health Telematics, 2008,
pp. 81–89.
[9] A. Cimatti, M. Roveri, and D. Sheridan, “Bounded verification of past
ltl,” in Proc. Int. Conf. Formal Methods Comput-Aided Design, 2004,
pp. 245–259.
[10] E. M. Clarke, E. A. Emerson, and A. P. Sistla, “Automatic verification of
finite-state concurrent systems using temporal logic specifications,” ACM
Trans. Program. Languages Syst., vol. 8, pp. 244–263, 1986.
[11] J. Coutaz, J. L. Crowley, S. Dobson, and D. Garlan, “Context is key,”
Commun. ACM, vol. 48, pp. 49–53, Mar. 2005.
[12] F. Cruciani, M. P. Donnelly, C. D. Nugent, G. Parente, and C. Paggetti,
“Dante: A video based annotation tool for smart environments,” in presented at the Int. Conf. Software Testing, Conf. Sensor Systems and Software, Miami, FL, USA, 2010.
[13] S. K. Das, D. J. Cook, A. Battacharya, III E. O. Heierman, and T.-Y. Lin,
“The role of prediction algorithms in the mavhome smart home architecture,” IEEE Wireless Commun., vol. 9, no. 6, pp. 77–84, Dec. 2002.
[14] L. De Alfaro, M. Faella, T. A. Henzinger, R. Majumdar, and M. Stoelinga,
“Model checking discounted temporal properties,” Theoretical Comput.
Sci., vol. 345, no. 1, pp. 139–170, 2005.
[15] M. P. Donnelly, T. Magherini, C. D. Nugent, F. Cruciani, and C. Paggetti,
“Annotating sensor data to detect activities of daily living,” in Proc. 9th
Int. Conf. Smart Homes Health Telematics, 2011, pp. 41–48.
[16] T. V. Duong, H. H. Bui, D. Q. Phung, and S. Venkatesh, “Activity recognition and abnormality detection with the switching hidden semimarkov
model,” in Proc. IEEE Comp. Soc. Conf. Comput. Vis. Pattern Recog.,
Jun. 2005, vol. 1, pp. 838–845.
[17] M. B. Dwyer, G. S. Avrunin, and J. C. Corbett, “Patterns in property specifications for finite-state verification,” in Proc. Int. Conf. Softw. Eng., 1999,
pp. 411–420.
[18] H. Hansson and B. Jonsson, “A logic for reasoning about time and reliability,” Formal Aspects Comput., vol. 6, no. 5, pp. 512–535, 1994.
[19] X. Hong, C. D. Nugent, M. Mulvenna, S. McClean, B. Scotney, and
S. Devlin, “Evidential fusion of sensor data for activity recognition in
smart homes,” Pervasive Mob. Comput., vol. 5, pp. 236–252, Jun. 2009.
[20] Y. A. Ivanov and A. F. Bobick, “Recognition of visual activities and interactions by stochastic parsing,” IEEE Trans. Pattern Anal. Mach. Intell.,
vol. 22, no. 8, pp. 852–872, Aug. 2000.
[21] D. N. Jansen, “More or less true DCTL for continuous-time MDPS,” in
Formal Modeling and Analysis of Timed Systems. New York, NY, USA:
Springer, 2013, pp. 137–151.
[22] R. Jhala and R. Majumdar, “Software model checking,” ACM Comput.
Surveys, vol. 41, no. 4, pp. 1–21, Article 21, 2009.
[23] E. M. Clarke Jr., O. Grumberg, and D. A. Peled, Model Checking. Cambridge, MA, USA: MIT Press, 1999.
[24] S. Katz, T. D. Downs, H. R. Cash, and R. C. Grotz, “Progress in development of the index of ADL,” The Gerontologist, vol. 10, no. 1 Part 1,
pp. 20–30, 1970.
[25] D. H. Kraft, G. Pasi, and G. Bordogna, “Vagueness and uncertainty in information retrieval: How can fuzzy sets help?” in Proc. 2006 Int. Workshop
Res. Issues Digital Libraries, New York, NY, USA, 2007, pp. 3:1–3:10.
[26] K. J. Kristoffersen, C. Pedersen, and H. R. Andersen, “Runtime verification of timed ltl using disjunctive normalized equation systems,” Electron.
Notes Theoretical Comput. Sci., vol. 89, no. 2, pp. 210–225, 2003.
[27] V. G. Kulkarni, Modeling and Analysis of Stochastic Systems. London,
U.K.: Chapman & Hall, 1995.
[28] T. Magherini, G. Parente, C. D. Nugent, M. P. Donnelly, E. Vicario,
F. Cruciani, and C. Paggetti, “Temporal logic bounded model-checking
for recognition of activities of daily living,” in Proc. 10th IEEE Int. Conf.
Inform. Technol. Appl. Biomed., 2010, pp. 1–4.
[29] S. Mckeever, J. Ye, L. Coyle, C. Bleakley, and S. Dobson, “Activity
recognition using temporal evidence theory,” J. Ambient Intell. Smart
Environ., vol. 2, pp. 253–269, Aug. 2010.
[30] A. Mihailidis, B. Carmichael, and J. Boger, “The use of computer vision
in an intelligent environment to support aging-in-place, safety, and independence in the home,” IEEE Trans. Inform. Technol. Biomed., vol. 8,
no. 3, pp. 238–247, Sep. 2004.
[31] S. P. Miller, M. W. Whalen, and D. D. Cofer, “Software model checking
takes off,” Commun. ACM, vol. 53, no. 2, pp. 58–64, Feb. 2010.
[32] C. D. Nugent, M. D. Mulvenna, X. Hong, and S. Devlin, “Experiences
in the development of a smart lab,” Int. J. Biomed. Eng. Technol., vol. 2,
pp. 319–331, 2009.
[33] S. A. ONeill, S. Mason, G. Parente, M. P. Donnelly, C. D. Nugent,
S. McClean, B. Scotney, and D. Craig, “Video reminders as cognitive
prosthetics for people with dementia,” Ageing Int., vol. 36, no. 2, pp. 267–
282, 2011.
[34] M. Philipose, K. P. Fishkin, M. Perkowitz, D. J. Patterson, D. Fox,
H. Kautz, and D. Hahnel, “Inferring activities from interactions with
objects,” IEEE Pervasive Comput., vol. 3, no. 4, pp. 50–57, Oct.–Dec.
2004.
[35] A. Rugnone, E. Vicario, C. D. Nugent, M. P. Donnelly, D. Craig,
C. Paggetti, and E. Tamburini, “Hometl: A visual formalism, based on
temporal logic, for the design of home based care,” in Proc. IEEE Int.
Conf. Autom. Sci. Eng., Sep. 2007, pp. 747–752.
MAGHERINI et al.: USING TEMPORAL LOGIC AND MODEL CHECKING IN AUTOMATED RECOGNITION OF HUMAN ACTIVITIES
[36] M. S. Ryoo and J. K. Aggarwal, “Recognition of composite human activities through context-free grammar based representation,” in Proc. IEEE
Comp. Soc. Conf. Comput. Vis. Pattern Recog., 2006, vol. 2, pp. 1709–
1718.
[37] M. S. Ryoo and J. K. Aggarwal, “Semantic representation and recognition
of continued and recursive human activities,” Int. J. Comput. Vis., vol. 82,
no. 1, pp. 1–24, 2009.
[38] A. Serna, H. Pigot, and V. Rialle, “Modeling the progression of alzheimer’s
disease for cognitive assistance in smart homes,” User Modeling UserAdapted Interaction, vol. 17, pp. 415–438, Sep. 2007.
[39] P. Smets, “Imperfect information: Imprecision, and uncertainty,” Uncertainty Manage. Inform. Syst., pp. 225–254, 1996.
[40] P. Turaga, R. Chellappa, V. S. Subrahmanian, and O. Udrea, “Machine
recognition of human activities: A survey,” IEEE Trans. Circuits Syst.
Video Technol., vol. 18, no. 11, pp. 1473–1488, Sep. 2008.
[41] D. L. Vail, M. M. Veloso, and J. D. Lafferty, “Conditional random fields
for activity recognition,” in Proc. 6th Int. Joint Conf. Auton. Agents Multiagent Syst., New York, NY, USA, 2007, pp. 235:1–235:8.
[42] T. van Kasteren, A. Noulas, G. Englebienne, and B. Krose, “Accurate
activity recognition in a home setting,” in Proc. 10th Int. Conf. Ubiquitous
Comput., 2008, pp. 1–9.
[43] C.-L. Wu and L.-C. Fu, “Design and realization of a framework for humansystem interaction in smart homes,” IEEE Trans. Syst., Man, Cybern. A,
Syst., Humans, vol. 42, no. 1, pp. 15–31, Jan. 2012.
[44] S. S. Yau, F. Karim, Y. Wang, and B. Wang, “Reconfigurable contextsensitive middleware for pervasive computing,” IEEE Pervasive Comput.,
vol. 1, no. 3, pp. 33–40, Jul.–Sep. 2002.
[45] J. Ye, S. Dobson, and S. McKeever, “Situation identification techniques in
pervasive computing: A review,” Pervasive Mobile Comput., vol. 8, no. 1,
pp. 36–66, 2012.
[46] X. Zhang, X. Chen, Y. Li, V. Lantz, K. Wang, and J. Yang, “A framework for hand gesture recognition based on accelerometer and emg sensors,” IEEE Trans. Syst., Man, Cybern., A, Syst., Humans, vol. 41, no. 6,
pp. 1064–1076, Nov. 2011.
Tommaso Magherini received the master’s degree in
informatics engineering in 2010 from the University
of Florence, Florence, Italy, where he is now completing the Ph.D. degree in informatics, multimedia,
and telecommunications engineering.
His research mainly focuses on formal methods
for pattern recognition.
521
Alessandro Fantechi received the Master degree in
computer science from the University of Pisa, Pisa,
Italy, in 1984, together with the Diploma di Licenza
from Scuola Normale Superiore.
He is a Full Professor with the University of Florence, Florence, Italy, where he is an active Researcher in the areas of formal development and verification of safety-critical systems, with a particular experience on railway signalling systems. He has
written more than 80 papers for international journals
and conferences, and maintains active collaborations
with industry in the sector of embedded safety-critical systems.
Chris D. Nugent (S’96–A’99–M’03) was born in
1973. He received the B.Eng. degree in electronic
systems and the D.Phil. degree in biomedical engineering from the University of Ulster, Jordanstown,
U.K., in 1995 and 1998, respectively.
In 1998, he took the post of Research Fellow with
the University of Ulster, London, U.K., and currently
holds the post of Professor of biomedical engineering.
His research interests include the design and evaluation of pervasive solutions in smart environments for
ambient-assisted living applications.
Enrico Vicario (M’94) received the master degree in
electronics engineering and the Ph.D. degree in informatics and telecommunications engineering from
the University of Florence, Florence, Italy, in 1990
and 1994, respectively.
He is a Full Professor of Computer Science at the
Department of Information Engineering of the University of Florence. His research is presently focused
on formal methods for model driven development,
correctness verification, and quantitative evaluation
of real-time concurrent systems.