Finite-state Machines: Theory and Applications
Unweighted Finite-state Automata
Thomas Hanneforth
Institut für Linguistik
Universität Potsdam
December 10, 2008
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
1 / 99
Overview
1
Applications of finite-state machines
2
Finite state acceptors and transducers: formal characterization
3
Closure properties and algebra of finite state acceptors
4
Closure properties and algebra of finite state transducers
5
Equivalence transformations on finite state acceptors
6
Equivalence transformations on finite state transducers
7
Decidability properties of unweighted finite-state acceptors and
transducers
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
2 / 99
Outline
1
Applications of Finite-State Machines
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
3 / 99
Some Applications of Finite-State Machines in
Computational Linguistics
Morphological analysis: lemmatization, word segmentation,
segmentation disambiguation
Spelling correction
Lexicon representation
Part-of-Speech-Tagging
Shallow parsing, Chunking
Speech recognition, speech synthesis
Optimality theory
Language modeling & Statistical language processing
Statistical machine translation
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
5 / 99
Outline
2
Finite state acceptors and transducers: formal characterization
Finite-state acceptors
Finite-state transducers
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
6 / 99
Finite state acceptors and transducers:
formal characterization
Finite-state acceptors
Non-deterministic finite-state acceptor
Definition (Non-deterministic finite-state acceptor (NFA))
A non-deterministic finite-state acceptor A is a 5-tuple hQ, Σ, q0 , F, δi where
Q is a non-empty set of states
Σ is a non-empty set and called the alphabet of A
q0 ∈ Q, the start state
F ⊆ Q, the set of final states
δ : Q × Σ ∪ {ε} 7→ 2Q , the transition function.
Notes
δ may be a partial function (and usually is)
Nondeterminism: the transition function δ maps a state q and an alphabet
symbol a to a set of successor states.
A transition may be labeled with ε, the neutral element of concatenation.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
8 / 99
Finite-state acceptors
Example (Nondeterministic FSA Alex accepting some animal names)
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
9 / 99
Finite-state acceptors
Deterministic finite-state acceptor
Definition (Deterministic finite-state acceptor (DFA))
A deterministic finite-state acceptor A is a 5-tuple hQ, Σ, q0 , F, δi, where
Q is a non-empty set of states
Σ is a non-empty set and called the alphabet of A
q0 ∈ Q, the start state
F ⊆ Q, the set of final states
δ : Q × Σ 7→ Q, the transition function.
Notes
Again, δ may be a partial function,
DFSA are by definition ε-free, that is, contain no ε-transitions.
DFSA and NDFA have the same generative power that is both concepts
are equivalent (cf. subset construction).
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
10 / 99
Finite-state acceptors
Example (Deterministic version of Alex )
Deterministic acyclic FSA are also called tries. Tries are useful for
lexicon representation.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
11 / 99
Finite-state acceptors
Examples
Example (DFSA AN P accepting English noun phrase patterns)
Example (DFSA AEnglish accepting some English sentences)
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
12 / 99
Finite-state acceptors
Generalized transition function, language
Definition (Generalized transition function δ ∗ )
δ ∗ is the reflexive and transitive closure of δ
δ ∗ (q, ε) = q, ∀q ∈ Q
δ ∗ (q, aw) = δ ∗ (δ(q, a), w)
Definition (Language of a DFSA A)
L(A) = {w ∈ Σ∗ | δ ∗ (q0 , w) ∈ F }
We also say that L(A) is recognized by A.
Definition (Regular language)
The language is called regular if there exists some DFA which recognizes it.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
13 / 99
Finite-state transducers
Definition
Definition ((Non-deterministic) finite-state transducer (NFST))
A (non-deterministic) finite-state transducer T is a 7-tuple
hQ, Σ, ∆, q0 , F, δ, σi, where
Q is a non-empty set of states
Σ is a non-empty set and called the input alphabet of T
∆ is a non-empty set and called the output alphabet of T
q0 ∈ Q, the start state
F ⊆ Q, the set of final states
δ : Q × Σ ∪ {ε} 7→ 2Q , the transition function.
σ : Q × Σ ∪ {ε} × Q 7→ ∆∗ , the output function.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
14 / 99
Finite-state transducers
Alternative definition
To simplify some definitions, we combine transition and output function
to a set of transitions.
In addition, we restrict the output function to single symbols or ε.
Definition (Normalized finite-state transducer)
A normalized finite-state transducer T is a 6-tuple hQ, Σ, ∆, q0 , F, Ei, where
Q is a non-empty set of states
Σ is a non-empty set, the input alphabet of T
∆ is a non-empty set, the output alphabet of T
q0 ∈ Q, the start state
F ⊆ Q, the set of final states
E ⊆ Q × (Σ ∪ {ε}) × (∆ ∪ {ε}) × Q, the set of transitions.
Note that every transducer can be transformed into a normalized
transducer.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
15 / 99
Finite-state transducers
Example (Tlex mapping surface forms to morph. features)
Note that fish is nondeterministically mapped to { NOUN sg, NOUN pl}
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
16 / 99
Finite-state transducers
Example (Laughter machine Tlaugh )
The input string laugh is mapped to the infinite set {han |n ≥ 1}
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
17 / 99
Finite-state transducers
Example (Bracketing machine Tbracket )
Every occurrence of ab is enclosed within brackets.
For example, the input string cabbabc is mapped to c{ab}b{ab}c by
traversing the state sequence 0 0 2 3 4 0 0 2 3 4 0 0
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
18 / 99
Finite-state transducers
Language
Definition (Language of a FST)
The language L(T ) of a FST T = hQ, Σ, ∆, q0 , F, δ, σi is defined in the
following way:
L(T ) = {hu, vi | δ ∗ (q0 , u) ∩ F 6= ∅ ∧ v ∈ σ ∗ (q0 , u)}
δ ∗ is recursively defined:
δ ∗ (q, ε) = {q} and
S
δ ∗ (q, wa) = q0 ∈δ∗ (q,w) δ(q 0 , a)
σ ∗ is the generalized output function and defined like this:
σ ∗ (q, ε) = {ε} and
σ ∗ (q, wa) = σ ∗ (q, w) ·
Thomas Hanneforth (Universität Potsdam)
S
q 0 ∈δ ∗ (q,w) σ(q
0 , a, p)
Finite-state Machines: Theory and Applications
December 10, 2008
19 / 99
Finite-state transducers
Transduction mapping
Definition (Transduction mapping)
The transduction mapping JT K : Σ∗ 7→ ∆∗ of a FST T is defined as:
JT K(x) = {y | hx, yi ∈ L(T )}
Definition (Functional transducer)
A transducer T is called functional if |JT K(x)| ≤ 1 for all x ∈ Σ∗
Example
Tbracket is functional. Tlex is not functional.
Definition (Ambiguous transducer)
A transducer T is called ambiguous, if |JT K(x)| > 1 for some x ∈ Σ∗ .
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
20 / 99
Finite-state transducers
Ambiguous transducers
Definition (Finitely ambiguous transducer)
A transducer T is called finitely ambiguous, if |JT K(x)| is finite for all x ∈ Σ∗ .
Example
Tlex is finitely ambiguous.
Definition (Infinitely ambiguous transducer)
A transducer T is called infinitely ambiguous, if |JT K(x)| is infinite for some
x ∈ Σ∗ .
Example
Tlaugh is infinitely ambiguous.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
21 / 99
Finite-state transducers
Deterministic finite-state transducer
Definition (Deterministic finite-state transducer (DFST))
A deterministic finite-state transducer T is a 7-tuple hQ, Σ, ∆, q0 , F, δ, σi,
where
Q is a non-empty set of states
Σ is a non-empty set and called the input alphabet of T
∆ is a non-empty set and called the output alphabet of T
q0 ∈ Q, the start state
F ⊆ Q, the set of final states
δ : Q × Σ 7→ Q, the (deterministic) transition function.
σ : Q × Σ 7→ ∆∗ , the (deterministic) output function.
Theorem
Every deterministic transducer is functional.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
22 / 99
Outline
3
Closure properties and algebra of finite state acceptors
Union
Concatenation
Star closure
Plus closure
Reversal
Complementation
Intersection
Difference
Substitution
Homomorphism
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
23 / 99
Closure properties and algebra
of finite state acceptors
Closure properties and algebra of finite state acceptors
Definition (Closure of a set)
Let S be a set and let fk be a k-ary function taking k-tuples over S as
arguments. We say that S is closed under fk if for all ai ∈ S
fk (a1 , a2 , . . . , ak ) ∈ S.
Note
Closure properties are important for the modularity based on a specific
formalism. They allow to build complex things out of simpler ones by
combining them with a number of operations.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
25 / 99
Closure properties and algebra of finite state acceptors
Closure properties of regular languages
The set of languages which is recognized by finite-state acceptors (the regular
languages) is closed under
Union
Concatenation
Plus and star closure
Reversal
Complementation
Intersection
Difference
Homomorphism and substitution
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
26 / 99
Closure properties and algebra of finite state acceptors
Union
Example (Union of two acceptors)
A2
A1
A1 ∪ A2
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
27 / 99
Closure properties and algebra of finite state acceptors
Concatenation
Example (Concatenation of two acceptors)
A2
A1
A1 · A2
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
28 / 99
Closure properties and algebra of finite state acceptors
Star closure
Example (Star (Kleene) closure of an acceptor)
A
A∗
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
29 / 99
Closure properties and algebra of finite state acceptors
Plus closure
Example (Plus closure of an acceptor)
A
A+
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
30 / 99
Closure properties and algebra of finite state acceptors
Reversal
Definition (Reversal of a string)
The reversal of a string w ∈ Σ∗ – denoted by wR – is defined as:
R = (a · w)R = wR · a, ∀a ∈ Σ ∧ w ∈ Σ∗
Example
obamaR = bamaR · o = amaR · bo = maR · abo = aR · mabo = · amabo
Definition (Reversal of a string set)
Let S ⊆ Σ∗ be a set of strings. The reversal of S - denoted by S R – is defined
as: S R = {wR |w ∈ S}.
Theorem
The set of regular languages is closed under reversal.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
31 / 99
Closure properties and algebra of finite state acceptors
Reversal
Example (Reversal of an acceptor)
A
AR
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
32 / 99
Closure properties and algebra of finite state acceptors
Complementation
Given an alphabet Σ and a FSA A you sometimes need a FSA A representing
all strings x over Σ∗ which are not in A.
Formally: L(A) = {x ∈ Σ∗ |x 6= L(A)} or L(A) = Σ∗ − L(A)
L(A)
L(A)
Theorem
The set of regular languages is closed under complementation.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
33 / 99
Closure properties and algebra of finite state acceptors
Complementation: algorithm
Algorithm:
1
2
3
Determinize A and obtain A0 .
Make A0 complete by adding a sink state s and adding for each state q
and each symbol a ∈ Σ not already used at q a transition δ(q, a) = s.
Exchange final and non-final states.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
34 / 99
Closure properties and algebra of finite state acceptors
Complementation
Example (Complementation of a finite-state acceptor)
A
(Σ = {a, c, r, t})
A
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
35 / 99
Closure properties and algebra of finite state acceptors
Complementation
Example (Why complementation works)
Consider a trie for W = {cat, camel, dog, f rog}.
Definition (Definition of a trie)
Let W be a finite set of words over Σ. Let P ref (W ) the set of all prefixes of
W . Define a DFA A = hP ref (W ), Σ, ε, W, δi with
∀a ∈ Σ, x, xa ∈ P ref (W ): δ(x, a) = xa.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
36 / 99
Closure properties and algebra of finite state acceptors
Complementation
Example (A trie for W = {cat, camel, dog, f rog})
States are labeled with prefixes of W .
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
37 / 99
Closure properties and algebra of finite state acceptors
Complementation
In a trie – a special acyclic DFA – each state corresponds to a single
prefix of a word in W .
In a general DFA A, each state q corresponds to a set of prefixes of the
words in L(A), the left language of q.
Definition (Left language)
←
−
The left language of a state q – symbolically L (q) – is defined as:
←
−
L (q) = {w ∈ Σ∗ | δ ∗ (q0 , w) = q}
Example (Left language)
←
−
L (1) = {a(ba)n | n ≥ 0}
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
38 / 99
Closure properties and algebra of finite state acceptors
Complementation: Why can we only complementize DFAs?
Example (’Complementation’ of an NFA)
The ”complementized”
NFA still accepts ab.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
39 / 99
Closure properties and algebra of finite state acceptors
Complementation: Importance
Complementation (negation) is important for the inherent robustness of
methods based on finite-state automata.
A NLP system is called robust if there does not exist an input string for
which it fails. That means: A robust NLP system accepts Σ∗ .
This is immediately related to negation: if there is some input string w
which is not accepted by some DFA A (say, for example, a DFA
representing some NP grammar about stock indices), one could use A to
accept w: L(A) ∪ L(A) = Σ∗ .
This property does not carry over to context-free languages: they are not
closed under complementation.
Context-sensitive languages are – perhaps surprisingly – again closed
under complementation. But their recognition problem is notoriously
difficult.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
40 / 99
Closure properties and algebra of finite state acceptors
Complementation: the sting in the tail: complexity
Theorem
Consider an NFA A with state set Q. The state complexity of an equivalent
DFA A0 can be in the worst case in O(|Σ|2|Q| ).
Example (Worst case of determinization)
Consider the regular language L = Σ∗ a(a|b)k for some k (with Σ = {a, b}).
While an NFA for L has k + 2 states, the equivalent DFA has 2k+1 states.
k=2
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
41 / 99
Closure properties and algebra of finite state acceptors
Intersection
If we know that FSAs are closed under complementation and union then
we also know that they are closed under intersection.
Why? By DeMorgan!
A∩B ≡A∪B
But this approach is very complex, since it requires three
complementation operations which in turn require determinization.
There is a more direct method: we let pair of states of the original FSAs
be states of the intersection FSA.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
42 / 99
Closure properties and algebra of finite state acceptors
Intersection: Example
Example (Intersection with the product state construction)
A1
A2
A1 ∩ A2
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
43 / 99
Closure properties and algebra of finite state acceptors
Intersection: formal definition
Definition (Intersection of two finite-state acceptors)
Let A1 = hQ1 , Σ1 , q01 , F1 , δ1 i and A2 = hQ2 , Σ2 , q02 , F2 , δ2 i be two FSAs.
A1 ∩ A2 , the intersection of A1 and A2 is an acceptor:
A = hQ1 × Q2 , Σ1 ∩ Σ2 , hq01 , q02 i, F1 × F2 , δi
where hp0 , q 0 i ∈ δ(hp, qi, a) if p0 ∈ δ1 (p, a) and q 0 ∈ δ2 (q, a) for all
a ∈ Σ1 ∩ Σ2 .
This mathematical approach generates in the worst as in the best case a
FSA with |Q1 ||Q2 | states.
But a lot of these states may not contribute to the language
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
44 / 99
Closure properties and algebra of finite state acceptors
Intersection: useless states
Definition (Inaccessible and non-coaccessible states)
Let A be a finite-state automaton (acceptor or transducer) with start state q0 .
A state q in A is called inaccessible if there is no path in A from q0 to q.
A state q in A is called non-coaccessible if there is no path in A from q to a
final state of A.
A state is called useless if it is inaccessible or non-coaccessible.
A finite-state automaton A is called trim or connected if it has no useless
states.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
45 / 99
Closure properties and algebra of finite state acceptors
Intersection: removal of useless states
Algorithm connect(A)
Require: FSM A with start state q0 , state set Q and final state set F
Ensure: A without useless states
1: Perform a depth-first search starting at q0 and mark each visited state
2: Delete each unmarked state q and all its ingoing and outgoing transitions
3: Reverse A
4: Unmark all states in Q
5: Perform a depth-first search starting at all states q ∈ F and mark each
visited state
6: Delete each unmarked state q and all its ingoing and outgoing transitions
7: Reverse A
Complexity of connect(A)
If A has |Q| number of states and |E| number of transitions, the complexity of
connect(A) is in O(|Q| + |E|).
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
46 / 99
Closure properties and algebra of finite state acceptors
Intersection: algorithm
Require: FSAs A1 = hQ1 , Σ1 , q01 , F1 , δ1 i and A2 = hQ2 , Σ2 , q02 , F2 , δ2 i
Ensure: A = A1 ∩ A2
1: F := Q := ∅
2: EN QU EU E(S, hq01 , q02 i)
3: while S 6= ∅ do
4:
hq1 , q2 i := DEQU EU E(S)
5:
for all a ∈ Σ1 ∩ Σ2 do
6:
if q10 ∈ δ1 (q1 , a) ∧ q20 ∈ δ2 (q2 , a) then
7:
δ(hq1 , q2 i, a) := δ(hq1 , q2 i, a) ∪ {hq10 , q20 i}
8:
if hq10 , q20 i ∈
/ Q then
9:
Q := Q ∪ {hq10 , q20 i}
10:
if q10 ∈ F1 ∧ q20 ∈ F2 then
11:
F := F ∪ {hq10 , q20 i}
12:
end if
13:
EN QU EU E(S, hq10 , q20 i)
14:
end if
15:
end if
16:
end for
17: end while
18: CON N ECT (A)
19: return A
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
47 / 99
Closure properties and algebra of finite state acceptors
Intersection: practical importance
Closure under intersection means that we can develop constraints
independently of each other and then enforce their validity
simultaneously by intersecting them
A lot of finite-state based NLP is based on intersection: (Two-level-)
Morphology, Constraint based grammar, pattern matching etc.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
48 / 99
Closure properties and algebra of finite state acceptors
Difference
Definition (Difference)
Let A1 and A2 two FSAs. The difference A1 − A2 is defined as:
A1 − A2 ≡ A1 ∩ A2
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
49 / 99
Closure properties and algebra of finite state acceptors
Difference
Example (Difference)
A2
A1
A1 − A2
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
50 / 99
Closure properties and algebra of finite state acceptors
Substitution
Definition (Substitution)
∗
A substitution is a mapping s : Σ 7→ 2∆ for two alphabets Σ and ∆.
∗
s is generalized to s∗ : Σ∗ 7→ 2∆ by:
s∗ (ε) = ε
s∗ (xa) = s∗ (x)s(a)
Theorem (Closure under substitution)
The set of regular languages is closed under substitution with regular
languages.
Note
A lot of finite-state based NLP is based on closure under substitution.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
51 / 99
Closure properties and algebra of finite state acceptors
Substitution
Example (Substitution in computational morphology)
A morphology rule as a FSA A
A2
A1
Result of the substitution A{N ST EM = A1 , N IN F L = A2 }
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
52 / 99
Closure properties and algebra of finite state acceptors
Homomorphism
Definition (Homomorphism)
A homomorphism is a mapping h : Σ 7→ ∆∗ for two alphabets Σ and ∆.
Definition (Inverse homomorphism)
Given a homomorphism h, the inverse homomorphic image h−1 of a language
L is defined as: h−1 (L) = {w | h(w) ∈ L}
Theorem (Closure under homomorphism)
The set of regular languages is closed under homomorphism and inverse
homomorphism
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
53 / 99
Outline
4
Closure properties and algebra of finite state transducers
Projection
Composition
Cross product
Inversion
Intersection
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
54 / 99
Closure properties and algebra
of finite state transducers
Closure properties and algebra of finite state transducers
The set of finite state transducers is closed under
Union
Concatenation
Closure
Reversal
Projection (note that this leads to FSAs)
Composition
Inversion
Finite state transducers are not closed under
Complementation
Intersection (but acyclic and ε-free transducers are)
Difference
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
56 / 99
Closure properties and algebra of finite state transducers
Projection
Definition (First and second projection)
Let T = hQ, Σ, ∆, q0 , F, Si be a transducer.
The first projection of T – symbolically π1 (T ) – is the FSA
A = hQ, Σ, q0 , F, δi where
∀a ∈ Σ ∪ {ε}, δ(p, a) = {q | ∃b ∈ ∆ : hp, a, b, qi ∈ S}
The second projection of T – symbolically π2 (T ) – is the FSA
A = hQ, Σ, q0 , F, δi where
∀b ∈ ∆ ∪ {ε}, δ(p, b) = {q | ∃a ∈ Σ : hp, a, b, qi ∈ S}
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
57 / 99
Closure properties and algebra of finite state transducers
Projection
Example (Projection)
Transducer T
π1 (T )
Thomas Hanneforth (Universität Potsdam)
π2 (T )
Finite-state Machines: Theory and Applications
December 10, 2008
58 / 99
Closure properties and algebra of finite state transducers
Composition
Composing a transducer T1 with a transducer T2 (formally T1 ◦ T2 ) means:
take some input u for T1 , collect the output v of T1 , feed it as input into T2
and collect the output w of T2 .
Definition (Composition relation)
Let T1 = hQ1 , Σ1 , ∆1 , q01 , F1 , S1 i and T2 = hQ2 , Σ2 , ∆2 , q02 , F2 , S2 i be
transducers. L(T1 ◦ T2 ) =
{hu, wi ∈ Σ∗1 × ∆∗2 | ∃v ∈ ∆∗1 ∩ Σ∗2 : hu, vi ∈ L(T1 ) ∧ hv, wi ∈ L(T2 )}
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
59 / 99
Closure properties and algebra of finite state transducers
Composition
Definition (ε-free composition)
Let T1 = hQ1 , Σ1 , ∆1 , q01 , F1 , E1 i and T2 = hQ2 , Σ2 , ∆2 , q02 , F2 , E2 i be
two normalized, ε-free FSTs. T1 ◦ T2 , the composition of T1 and T2 , is the
transducer
T = hQ1 × Q2 , Σ1 , ∆2 , hq01 , q02 i, F1 × F2 , Ei where
E = {hhp, qi, a, b, hp0 , q 0 ii | ∃c ∈ ∆1 ∩ Σ2 :
hp, a, c, p0 i ∈ E1 ∧ hq, c, b, q 0 i ∈ E2 }
Properties of composition
The composition operation is not commutative, that is, in general:
T1 ◦ T2 6= T2 ◦ T1
The composition operation is associative, that is:
T1 ◦ T2 ◦ T3 = (T1 ◦ T2 ) ◦ T3 = T1 ◦ (T2 ◦ T3 )
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
60 / 99
Closure properties and algebra of finite state transducers
Composition
How does composition work?
Whenever T1 contains a transition:
and T2 contains a transition:
T will contain a transition:
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
61 / 99
Closure properties and algebra of finite state transducers
Composition
Example (Composition)
◦
=
FST mapping NP-patterns to
FST repeatedly mapping
NP category
words to their categories
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
62 / 99
Closure properties and algebra of finite state transducers
Application
Definition (Identity transducer)
Let A = hQ, Σ, q0 , F, δi be a FSA.
The identity transducer ID(A) is defined by hQ, Σ, Σ, q0 , F, Ei where
E = {hp, a, a, qi | ∃p, q ∈ Q, a ∈ Σ ∪ {ε} : q ∈ δ(p, a)}
Example (Identity transducer)
Definition (Application)
The application of a FST T to a FSA A – symbolically T [A] – is defined as
T [A] ≡ π2 (ID(A) ◦ T )
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
63 / 99
Closure properties and algebra of finite state transducers
Application
Example
FSA A
FSA ID(A)
FST T
ID(A) ◦ T
Thomas Hanneforth (Universität Potsdam)
T [A] = π2 (ID(A) ◦ T )
Finite-state Machines: Theory and Applications
December 10, 2008
64 / 99
Closure properties and algebra of finite state transducers
Composition: relationship to intersection
Composition can be considered as a generalization of intersection. The
intersection of two FSAs A1 and A2 can be defined as follows:
A1 ∩ A2 = π1 (ID(A1 ) ◦ ID(A2 ))
So, intersecting two FSAs is done by composing their identity transducers and
afterwards projecting one of the tapes. Composing two transducers X and Y
means synchronizing (intersecting) their inner tapes and then combining the
outer tapes:
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
65 / 99
Closure properties and algebra of finite state transducers
Composition: handling ε-transitions
It is possible to generalize the composition definition to transducers with
-transitions:
Definition (Transducer composition)
Let T1 = hQ1 , Σ1 , ∆1 , q01 , F1 , E1 i and T2 = hQ2 , Σ2 , ∆2 , q02 , F2 , E2 i be
two normalized FSTs.
T1 ◦ T2 , the composition of T1 and T2 , is the transducer
T = hQ1 × Q2 , Σ1 , ∆2 , hq01 , q02 i, F1 × F2 , E ∪ Eε ∪ Ei,ε ∪ Eo,ε i
where
1
{hhp, qi, a, b, hp0 , q 0 ii
E=
| ∃c ∈ ∆1 ∩ Σ2 :
hp, a, c, p0 i ∈ E1 ∧ hq, c, b, q 0 i ∈ E2 }
2
Eε = {hhp, qi, a, b, hp0 , q 0 ii | hp, a, ε, p0 i ∈ E1 ∧ hq, ε, b, q 0 i ∈ E2 }
3
Ei,ε = {hhp, qi, ε, a, hp, q 0 ii | hq, ε, a, q 0 i ∈ E2 ∧ p ∈ Q1 }
4
Eo,ε = {hhp, qi, a, ε, hp0 , qii | hp, a, ε, p0 i ∈ E1 ∧ q ∈ Q2 }
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
66 / 99
Closure properties and algebra of finite state transducers
Composition: handling ε-transitions
There are four different ways, how ε and alphabet symbols on the second tape
of T1 and the first tape of T2 can interact:
1 T contains a a : c-transition and T contains a c : b-transition: this is
1
2
handled in the same way as in the ε-free case
2 T contains a a : ε-transition and T contains a ε : b-transition →
1
2
T contains a a : b-transition. That is: ε is treated as a regular symbol.
3 T “stays” in the same state, T moves on:
1
2
4
T1 moves on, T2 “stays” in the same state:
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
67 / 99
Closure properties and algebra of WFSA
Composition
Example (Composition of two unweighted FSTs)
T2
T1
T1 ◦ T2
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
68 / 99
Closure properties and algebra of finite state transducers
Composition: Application
Composition is a very important operation for building processing or
filtering cascades, for example in robust parsing and morphological
analysis.
Since composition is not commutative, the order of a transducer cascade
C = T1 ◦ T2 ◦ . . . ◦ Tk matters.
This may lead to problems related to feeding, counter-feeding, bleeding
and counter-bleeding.
Since the composition operation is associative, the order in which the
compositions in C are computed does not matter. This entails some
freedom degrees for implementing such cascades.
Note, that the state complexity of T1 ◦ T2 ◦ . . . ◦ Tk is |Q1 ||Q2 | . . . |Qk |
in the worst case.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
69 / 99
Closure properties and algebra of finite state transducers
Cross product
Definition ((Cartesian) Product)
Given two sets S1 and S2 , the Cartesian product S1 × S2 is defined as:
S1 × S2 = {hx, yi | x ∈ S1 ∧ y ∈ S2 }
Theorem (Product of regular sets)
Let A1 = hQ1 , Σ1 , q01 , F1 , δ1 i and A2 = hQ2 , Σ2 , q02 , F2 , δ2 i be two
finite-state acceptors. Then L(A1 ) × L(A2 ) is representable by a finite-state
transducer A1 × A2 .
Proof.
A1 × A2 ≡ ID(A1 ) ◦ TΣ∗1 7→ε ◦ Tε7→Σ∗2 ◦ ID(A2 )
Note
Cross product is the core of all replacement operations.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
70 / 99
Closure properties and algebra of finite state transducers
Cross product
Example (Cross product)
A1
A2
TΣ∗1 7→ε
Tε7→Σ∗2
A1 × A2
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
71 / 99
Closure properties and algebra of finite state transducers
Inversion
Definition (Inversion)
Let T = hQ, Σ, ∆, q0 , F, Ei be a transducer.
The inversion of T – symbolically T −1 – is the FST
T −1 = hQ, ∆, Σ, q0 , F, E −1 i where E −1 = {hp, b, a, qi|hp, a, b, qi ∈ E}
Note
Thus, inversion simply exchanges input- and output “tapes” of a transducer.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
72 / 99
Closure properties and algebra of finite state transducers
Inversion
Example (Morphological analysis vs. generation)
FST TM orph mapping words to morphological categories
−1
FST TM
orph mapping morphological categories to words
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
73 / 99
Closure properties and algebra of finite state transducers
Why are FSTs not closed under intersection?
Example
Tan 7→bn c∗
Tan 7→b∗ cn
The intersection of Tan 7→bn c∗ and Tan 7→b∗ cn would result in the relation
R = {han , bn cn i | n ≥ 0} which is not regular and thus not representable by
a finite-state transducer.
Note
This has consequences for creating applications based on finite-state
transducers. They cannot be based on the intersection of constraints
represented as transducers.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
74 / 99
Closure properties and algebra of finite state transducers
Why are FSTs not closed under intersection?
Intuitively, the existence of ε within loops leading to infinite ambiguity is
the reason why FSTs are not closed under intersection
Thus, ε-free FSTs – also called equal-length transducers – are closed
under intersection
The same is true for acyclic FSTs, where we have some freedom where
to realize the ε-transitions
By DeMorgan, non-closure under intersection leads to non-closure under
complementation
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
75 / 99
Outline
5
Equivalence transformations on finite-state acceptors
ε-Removal
Determinization
Minimization
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
76 / 99
Equivalence transformations on finite-state
acceptors
Equivalence transformations on finite-state acceptors
Equivalence transformations
Equivalence transformations are operations on automata which change
the topology of an automaton without changing its language.
They usually serve optimization purposes, that is, they create smaller
and/or faster automata.
Finite-state acceptors admit the following equivalence transformations:
ε-Removal
Determinization
Minimization
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
78 / 99
Equivalence transformations on finite-state acceptors
ε-Removal
Definition ( )
Let p and q be states in Q and let w be a string in Σ∗ . Let w be a relation
Q × Q, such that hp, qi ∈ w if there is a path labeled with w from p to q.
Definition (ε-closure)
Given a NFA A = hQ, Σ, q0 , F, δi, ε-closure(q) = {q} ∪ {p ∈ Q | q
ε
p}.
Definition (ε-free FSA)
Let A = hQ, Σ, q0 , F, δi be a FSA. Define A0 , the equivalent ε-free FSA with
L(A0 ) = L(A), as A0 = hQ, Σ, q0 , F 0 , δ 0 i where:
S
δ 0 (q, a) = ε-closure( q0 ∈ε-closure(q) δ(q 0 , a)), ∀q ∈ Q, a ∈ Σ
F 0 = F ∪ {q0 }, if ε-closure(q0 ) ∩ F 6= ∅, else F 0 = F .
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
79 / 99
Equivalence transformations on finite-state acceptors
ε-Removal
Example
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
80 / 99
Equivalence transformations on finite-state acceptors
Determinization
Definition (Subset construction)
Let A = hQ, Σ, q0 , F, δi be a FSA. Define A0 , the equivalent DFA with
L(A0 ) = L(A) as A0 = h2Q , Σ, {q0 }, F 0 , δ 0 i with :
F 0 = {S ⊆ Q | S ∩ F 6= ∅}
S
δ 0 (S, a) = q∈S δ(q, a), ∀a ∈ Σ, ∀S ⊆ Q
Note
The complexity of an algorithm which implements this in a naive way is
exponential.
In the normal case, most of the subset states in the DFA are not
accessible / coaccessible.
A better algorithm based on a state queue avoids inaccessible states.
But this doesn’t change the complexity in the worst case.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
81 / 99
Equivalence transformations on finite-state acceptors
Determinization
Example
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
82 / 99
Equivalence transformations on finite-state acceptors
Minimization
Definition (Minimal DFA)
Let A = hQ, Σ, q0 , F, δi be a DFA.
0
L is minimal if ∀A0 = hQ0 , Σ0 , q0 , F 0 , δ 0 i : L(A0 ) = L(A) ⇒ |Q| ≤ |Q0 |
Definition (Right language)
Let A = hQ, Σ, q0 , F, δi be a DFA.
→
−
The right language of a state q ∈ Q – symbolically L (q) – is defined as:
→
−
L (q) = {w ∈ Σ∗ | δ ∗ (q, w) ∈ F }
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
83 / 99
Equivalence transformations on finite-state acceptors
Minimization
Definition (Equivalent states)
→
−
→
−
Two states p and q are called equivalent if L (p) = L (q).
This holistic definition based on right languages can be turned into a recursive
definition of equivalence of states:
Definition (State equivalence ≡)
Let A = hQ, Σ, q0 , F, δi be a DFA.
Two states p and q are called equivalent – symbolically p ≡ q –, if:
p ≡ q if p ∈ F ⇔ q ∈ F ∧ ∀a ∈ Σ : δ(p, a) ≡ δ(q, a).
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
84 / 99
Equivalence transformations on finite-state acceptors
Minimization
Based on state equivalence, we come up with a definition of a minimal DFA:
Theorem (Minimal DFA I)
Let A = hQ, Σ, q0 , F, δi be a DFA. A is minimal iff
→
−
→
−
∀p, q ∈ Q : p 6= q ⇒ L (p) 6= L (q).
By substituting the recursive definition of state equivalence into the last
theorem, we arrive at:
Theorem (Minimal DFA II)
Let A = hQ, Σ, q0 , F, δi be a DFA. A is minimal iff
∀p, q ∈ Q : p 6= q ⇒ p ∈ F < q ∈ F ∨ ∃a ∈ Σ : δ(p, a) 6≡ δ(q, a).
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
85 / 99
Equivalence transformations on finite-state acceptors
Minimization: Myhill-Nerode theorem
Theorem (Myhill-Nerode)
The following propositions are equivalent:
1
L is recognized by a DFA AL = hQ, Σ, q0 , F, δi.
2
L is the union of some equivalence classes of a right invariant
equivalence relation R with finite index.
3
RL (x RL y iff ∀z ∈ Σ∗ : xz ∈ L ⇔ yz ∈ L) is of finite index.
Notes
The Myhill-Nerode theorem links states with subsets of Σ∗ . It is central to the theorem that the
number of states in a DFA is finite.
The Myhill-Nerode theorem assumes complete DFAs, that is, the corresponding transition function
δ is total.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
86 / 99
Equivalence transformations on finite-state acceptors
Minimization: Myhill-Nerode theorem
Definition (Equivalence relation, equivalence class)
Let S be a set and E ⊆ S × S a binary relation. E is called a equivalence
relation if E is reflexive, symmetric and transitive.
If E is a equivalence relation, we call [x]E = {y | x E y} the equivalence
class of x wrt E.
Properties of equivalence relations
1
x ∈ [x]E , ∀x ∈ S
2
[x]E = [y]E ∨ [x]E ∩ [y]E = ∅, ∀x, y ∈ S
S
[x]E = S
3
x∈S
Definition (Index of a equivalence relation)
Let E be a equivalence relation. The index IE of E is the number of E’s
equivalence classes. E is of finite index if IE is finite.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
87 / 99
Equivalence transformations on finite-state acceptors
Minimization: Myhill-Nerode theorem
Definition (Right-invariant equivalence relation)
Let R be a equivalence relation over Σ∗ .
R is called right-invariant (with respect to concatenation) if
∀x, y, z ∈ Σ∗ : x R y ⇒ xz R yz
Definition (Left language)
←
−
Let A = hQ, Σ, q0 , F, δi be a DFA. Define the left language L (p) of a state
←
−
p ∈ Q as L (p) = {w ∈ Σ∗ | δ(q0 , w) = p}.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
88 / 99
Equivalence transformations on finite-state acceptors
Minimization: Myhill-Nerode theorem
Myhill-Nerode theorem.
We prove the theorem by chaining 1 ⇒ 2, 2 ⇒ 3 and 3 ⇒ 1.
1 ⇒ 2.
Let A be a DFA recognizing L.
Define RA as x Ra y if δ(q0 , x) = δ(q0 , y), ∀x, y ∈ Σ∗ .
Subproof: RA is right-invariant equivalence relation of finite index (the index
of RA is |Q|). The equivalence classes of RA are the left languages
S ←
←
−
−
L (p), ∀p ∈ Q.
L (q) = L.
q∈F
Example
←
−
←
−
←
−
L (1) = {ε, a(ba)∗ b}, L (2) = {a(ba)∗ }, L (3) = {a(ba)∗ c(d)∗ }
←
−
←
−
L = L (2) ∪ L (3)
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
89 / 99
Equivalence transformations on finite-state acceptors
Minimization: Myhill-Nerode theorem
Myhill-Nerode theorem (continued).
2 ⇒ 3.
R = RA is an refinement of RL , that is, every equivalence class of RA is
contained in some equivalence class of RL .
1
Assume that x RA y.
2
Since RA is right-invariant, xz RA yz, for all z ∈ Σ∗ .
3
Thus xz ∈ L if and only if yz ∈ L.
4
Thus xz RL yz and the equivalence class of x wrt RA is contained in the
equivalence class of x wrt RL .
5
Since the index of RA is finite (at most equal to |Q|) and the index of RL
is less or equal to the index of RA , we conclude that RL is of finite index.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
90 / 99
Equivalence transformations on finite-state acceptors
Minimization: Myhill-Nerode theorem
Example (RA is a refinement of RL )
State
3
9
15
17
19
22
24
Corresponding equiv. class of RA
e
frie
dog
doll
dollar
coll
collar
Thomas Hanneforth (Universität Potsdam)
State
7
8
12
16
Corresponding equiv. class of RL
dog, collar, dollar, end, friend, frog
e, frie
doll
coll
Finite-state Machines: Theory and Applications
December 10, 2008
91 / 99
Equivalence transformations on finite-state acceptors
Minimization: Myhill-Nerode theorem
Myhill-Nerode theorem (continued).
3 ⇒ 1.
Given RL , construct a new FSA A0 = hQ0 , Σ, q00 , F 0 , δ 0 i as follows:
1
Q0 = {[x] | [x] is a equivalence class of Σ∗ under RL }
2
q00 = [ε]
3
δ 0 : Q0 × Σ 7→ Q0 : δ 0 ([x], a) = [xa], ∀[x] ∈ Q0 ∧ a ∈ Σ
4
F 0 = {[x] | x ∈ L}
Example
δ 0 ([e], n) = [en]
δ 0 ([en], d) = [end]
a
δ 0 ({f rie, e}, n) = {f rien, en}
δ 0 ({f rien, en}, d) = {f riend, end}a
Inspired by CAKE: “Friend is a four-letter word”
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
92 / 99
Equivalence transformations on finite-state acceptors
Minimization: algorithms
Approaches
1
2
→
−
→
−
Union-Find-based: Find all states p and q with L (p) = L (q) and
merge them.
Partition-based: Starting at sets of non-equivalent states, partition these
sets further until each set contains only equivalent states.
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
93 / 99
Outline
6
Equivalence transformations on finite-state transducers
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
94 / 99
Equivalence transformations on finite-state
transducers
Outline
7
Decidability properties of unweighted finite-state acceptors and
transducers
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
96 / 99
Decidability properties of unweighted
finite-state acceptors and transducers
Decidability properties of unweighted finite-state acceptors
and transducers
Given two finite-state acceptors A and A0 , the following properties are
decidable:
L(A) = ∅
L(A) = Σ∗
L(A) = L(A0 )
L(A) ⊆ L(A0 )
Given two finite-state transducers T and T 0 , the following properties are
decidable:
T is functional
Given two finite-state transducers T and T 0 , the following properties are
undecidable:
L(T ) = L(T 0 )
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
98 / 99
Version history
16.10.08: version 0.1 (initial version)
20.10.08: version 0.2 (some error corrections, added definition of
composition of FSTs with ε-transitions)
09.11.08: version 0.3 (added example for ε-composition, enhanced
example for cross product, added subtitles)
07.12.08: version 0.4 (completed minimization section)
Thomas Hanneforth (Universität Potsdam)
Finite-state Machines: Theory and Applications
December 10, 2008
99 / 99