Theory of Languages
and Automata
Chapter 0 - Introduction
Sharif University of Technology
References
O Main Reference
 M. Sipser, ”Introduction to the Theory of Computation,” 2nd Ed.,
Thompson Learning Inc., 2006.
O Additional References
 P. Linz, “An Introduction to Formal Languages and Automata,” 3rd
Ed., Jones and Barlett Publishers, Inc., 2001.
 J.E. Hopcroft, R. Motwani and J.D. Ullman, “Introduction to
Automata Theory, Languages, and Computation,” 2nd Ed.,
Addison-Wesley, 2001.
 P.J. Denning, J.B. Dennnis, and J.E. Qualitz, “Machines,
Languages, and Computation,” Prentice-Hall, Inc., 1978.
 P.J. Cameron, “Sets, Logic and Categories,” Springer-Verlag,
London limited, 1998.
Theory of Languages and Automata
Prof. Movaghar
2
Grading Policy
O Assignments
O First Quiz
O Second Quiz
O Final Exam
Theory of Languages and Automata
30%
20%
20%
30%
Prof. Movaghar
3
Main Topics
O Complexity Theory
O Computability Theory
O Automata Theory
O Mathematical Notions
O Alphabet
O Strings
O Languages
Theory of Languages and Automata
Prof. Movaghar
4
Complexity Theory
O The scheme for classifying problems according to their
computational difficulty
O Options:
 Alter the problem to a more easily solvable one by
understanding the root of difficulty
 Approximate the perfect solution
 Use a procedure that occasionally is slow but usually
runs quickly
 Consider alternatives, such as randomized computation
O Applied area:
 Cryptography
Theory of Languages and Automata
Prof. Movaghar
5
Computability Theory
O
Determining whether a problem is solvable by
Computers
O Classification of problems as solvable ones and
unsolvable ones
Theory of Languages and Automata
Prof. Movaghar
6
Automata Theory
O Definitions and properties of mathematical models
of computation
 Finite automaton
Usage: Text Processing, Compilers, Hardware Design
 Context-free grammar
Usage: Programming Languages, Artificial Intelligence
Theory of Languages and Automata
Prof. Movaghar
7
Logic
O
The science of the formal principles of
reasoning
Theory of Languages and Automata
Prof. Movaghar
8
History of Logic
O Logic was known as 'dialectic' or 'analytic' in
Ancient Greece. The word 'logic' (from the Greek
logos, meaning discourse or sentence) does not
appear in the modern sense until the commentaries
of Alexander of Aphrodisias, writing in the third
century A.D.
O While many cultures have employed intricate
systems of reasoning, and logical methods are
evident in all human thought, an explicit analysis of
the principles of reasoning was developed only in
three traditions: those of China, India, and Greece.
Theory of Languages and Automata
Prof. Movaghar
9
History of Logic (cont.)
O Although exact dates are uncertain, particularly in the case of
India, it is possible that logic emerged in all three societies by
the 4th century BC.
O The formally sophisticated treatment of modern logic
descends from the Greek tradition, particularly Aristotelian
logic, which was further developed by Islamic Logicians and
then medieval European logicians.
O The work of Frege in the 19th century marked a radical
departure from the Aristotlian leading to the rapid
development of symbolic logic, later called mathematical
logic.
Theory of Languages and Automata
Prof. Movaghar
10
Modern Logic
O Descartes proposed using algebra, especially
techniques for solving for unknown quantities in
equations, as a vehicle for scientific exploration.
O The idea of a calculus of reasoning was also
developed by Leibniz. He was the first to formulate
the notion of a broadly applicable system of
mathematical logic.
O Frege in his 1879 work extended formal logic
beyond propositional logic to include quantification
to represent the "all", "some" propositions of
Aristotelian logic.
Theory of Languages and Automata
Prof. Movaghar
11
Modern Logic (cont.)
O A logic is a language of formulas.
O A formula is a finite sequence of symbols with a
syntax and semantics.
O A logic can have a formal system.
O A formal system consists of a set of axioms and
rules of inference.
Theory of Languages and Automata
Prof. Movaghar
12
Logic Types of Interest
O Propositional Logic
O Predicate Logic
Theory of Languages and Automata
Prof. Movaghar
13
Syntax of Propositional Logic
O Let {p0, p1, ..} be a countable set of
O
O
O
O
propositional variables,
{, , ¬, →, ↔} be a finite set of
connectives,
Also, there are left and right brackets,
A propositional variable is a formula,
If φ and ψ are formulas, then so are (¬φ), (φ  ψ),
(φ  ψ), (φ → ψ), and (φ ↔ ψ).
Theory of Languages and Automata
Prof. Movaghar
14
Semantics of Propositional Logic
O Any formula, which involves the propositional
variables p0,...,pn , can be used to define a function
of n variables, that is, a function from the set {T, F}n
to {T, F}. This function is often represented as the
truth table of the formula and is defined to be the
semantics of that formula.
Theory of Languages and Automata
Prof. Movaghar
15
Examples
ψ
φ
(¬φ)
(φ  ψ) (φ  ψ) (φ → ψ)
T
T
F
T
T
T
T
F
T
F
F
T
F
F
T
F
T
F
T
T
F
F
F
T
F
F
T
T
Theory of Languages and Automata
(φ ↔
ψ)
Prof. Movaghar
16
Other Definitions
O A formulae is a tautology if it is always true.
 (P  (¬P)) is a tautology.
O A formulae is a contradiction if it is never true.
 (P  (¬P)) is a contradiction.
O A formulae is a contingency if it is sometimes true.
 (P → (¬P)) is a contingency.
Theory of Languages and Automata
Prof. Movaghar
17
Formal System
A formal system includes the following:
O An alphabet A, a set of symbols.
O A set of formulae, each of which is a string of
symbols from A.
O A set of axioms, each axiom being a formula.
O A set of rules of inference, each of which takes as
„input‟ a finite sequence of formulae and produces as
output a formula.
Theory of Languages and Automata
Prof. Movaghar
18
Proof
O A proof in a formal system is just a finite
sequence of formulae such that each formula
in the sequence either is an axiom or is
obtained from earlier formulae by applying a
rule of inference.
Theory of Languages and Automata
Prof. Movaghar
19
Theorem
O A theorem of the formal system is just the
last formula in a proof.
O Example:
For any formula φ, the formula (φ→φ) is a
theorem of the propositional logic.
Theory of Languages and Automata
Prof. Movaghar
20
A Formal System for Propositional
Logic
O There are three „schemes‟ of axioms, namely:
 (A1) (φ→(ψ → φ))
 (A2) (φ→(ψ → θ)) → ((φ→ ψ) → (φ → θ))
 (A3) (((¬φ) →(¬ψ)) → (ψ → φ))
O Each of these formulas is an axiom, for all choices
of formulae φ, ψ, θ.
O There is only one rule of inference, namely Modus
Ponens: From φ and (φ→ ψ), infer ψ.
Theory of Languages and Automata
Prof. Movaghar
21
Example of a Proof
O Using (A2), taking φ, ψ, θ to be φ, (φ→φ) and φ
respectively
((φ→((φ→φ)→φ))→((φ→(φ→φ))→(φ →φ)))
O Using (A1), taking φ, ψ to be φ and (φ→φ) respectively
(φ→((φ→φ)→φ))
O Using Modus Ponens
((φ→(φ→φ))→(φ →φ))
O Using (A1), taking φ, ψ to be φ and φ respectively
(φ→(φ→φ))
O Using Modus Ponens
(φ→φ)
Theory of Languages and Automata
Prof. Movaghar
22
Soundness & Completeness
O A formal system is said to be sound if all
theorems in that system are tautology.
O A formal system is said to be complete if all
tautologies in that system are theorems.
Theory of Languages and Automata
Prof. Movaghar
23
Predicate
O A proposition involving some variables,
functions and relations
O Example
P(x) = “x > 3”
Q(x,y,z) = “x2 + y2 = z2”
Theory of Languages and Automata
Prof. Movaghar
24
Quantifier
∀
∀x P(x) ⇔ P(x1) ∧ P(x2) ∧ P(x3) ∧ …
O Universal: “for all”
O Existential: “there exists” ∃
∃x P(x) ⇔ P(x1) ∨ P(x2) ∨ P(x3) ∨ …
O Combinations:
∀x ∃y y > x
Theory of Languages and Automata
Prof. Movaghar
25
Quantifiers: Negation
O ¬ (∀x P(x)) ⇔ ∃x ¬ P(x)
O ¬ (∃x P(x)) ⇔ ∀x ¬ P(x)
O ¬ ∃x ∀y P(x,y) ⇔ ∀x ∃y ¬ P(x,y)
O ¬ ∀x ∃y P(x,y) ⇔ ∃x ∀y ¬ P(x,y)
Theory of Languages and Automata
Prof. Movaghar
26
Rules of Inference
1. Direct Proof
2. Indirect Proof
3. Proof by Contradiction
4. Proof by Cases
5. Induction
Theory of Languages and Automata
Prof. Movaghar
27
Direct Proof
O If the two propositions (premises) p and p → q are
theorems, we may deduce that the proposition q is
also a theorem.
O This fundamental rule of inference is called modus
ponens by logicians.
p→q
p
∴q
Theory of Languages and Automata
Prof. Movaghar
28
Indirect Proof
O Proves p → q by instead proving the contra-
positive, ~q → ~p
p→q
~q
∴ ~p
Theory of Languages and Automata
Prof. Movaghar
29
Proof by Contradiction
O The rule of inference used is that from
theorems p and ¬q → ¬p, we may deduce
theorem q.
p
~q → ~p
∴q
Theory of Languages and Automata
Prof. Movaghar
30
Proof by Cases
You want to prove p → q
O P can be decomposed to some cases:
p ↔ p1 ν p2 ν …ν pn
O
O Independently prove the n implications given
by
pi → q for 1 ≤ i ≤ n.
Theory of Languages and Automata
Prof. Movaghar
31
Proof by Induction
O To prove
xP (x )
Proof P(0),
2. Proof
x [P (x )  P (x 1)]
1.
Theory of Languages and Automata
Prof. Movaghar
32
Constructive Existence Proof
O Want to prove that
 x P(x)
O Find an a and then prove that P(a) is true
Theory of Languages and Automata
Prof. Movaghar
33
Non-Constructive Existence Proof
O Want to prove that
 x P(x)
O You cannot find an a that P(a) is true
O Then, you can use proof by contradiction:
~  x P( x)   x ~ P( x)  F
Theory of Languages and Automata
Prof. Movaghar
34
Intuitive (Naïve) Set Theory
There are three basic concepts in set theory:
O Membership
O Extension
O Abstraction
Theory of Languages and Automata
Prof. Movaghar
35
Membership
O
Membership is a relation that holds between a set
and an object
O x∈A to mean “the object x is a member of the set A”,
or “x belongs to A”. The negation of this assertion
is written x∉A as an abbreviation for the proposition
¬(x∈A)
O One way to specify a set is to list its elements. For
example, the set A = {a, b, c} consists of three
elements. For this set A, it is true that a∈A but d∉A
Theory of Languages and Automata
Prof. Movaghar
36
Extension
O The concept of extension is that two sets are
identical if and if only if they contain the
same elements. Thus we write A=B to mean
∀x [x∈A ⟺ x∈B]
Theory of Languages and Automata
Prof. Movaghar
37
Abstraction
Each property defines a set, and each set defines a
property
O If p(x) is a property then we can define a set A
A = {x ∣ p(x)}
O If A is a set then we can define a predicate p(x)
p(x) = ∀x (x∈A)
Theory of Languages and Automata
Prof. Movaghar
38
Intuitive versus axiomatic set theory
O The theory of set built on the intuitive concept of
membership, extension, and abstraction is known as
intuitive (naïve) set theory.
O As an axiomatic theory of sets, it is not entirely
satisfactory, because the principle of abstraction
leads to contradictions when applied to certain
simple predicates.
Theory of Languages and Automata
Prof. Movaghar
39
Russell‟s Paradox
Let p(X) be a predicate defined as
P(X) = (XX)
Define set R as
R={X|P(X)}
O Is P(R) true?
O Is P(R) false?
Theory of Languages and Automata
Prof. Movaghar
40
Gottlob Frege‟s Comments
O The logician Gottlob Frege was the first to develop
mathematics on the foundation of set theory. He
learned of Russell Paradox while his work was in
press, and wrote, “A scientist can hardly meet with
anything more undesirable than to have the
foundation give way just as the work is finished. In
this position I was put by a letter from Mr. Bertrand
Russell as the work was nearly through the press.”
Theory of Languages and Automata
Prof. Movaghar
41
Power Set
O The set of all subsets of a given set A is
known as the power set of A, and is denoted
by P(A):
P(A) = {B | B  A}
Theory of Languages and Automata
Prof. Movaghar
42
Set Operation-Union
O The union of two sets A and B is
A  B = {x | (x  A)  (x  B)}
and consists of those elements in at least one of A
and B.
O If A1, …, An constitute a family of sets, their union
is
= (A1  …  An)
= {x| x  Ai for some i, 1≤ i ≤ n}
Theory of Languages and Automata
Prof. Movaghar
43
Set Operation-Intersection
O The intersection of two sets A and B is
A  B = {x | (x  A) ∧ (x  B)}
and consists of those elements in at least one of A
and B.
O If A1, …, An constitute a family of sets, their
intersection is
= (A1  …  An)
= {x| x  Ai for all i, 1≤ i ≤ n}
Theory of Languages and Automata
Prof. Movaghar
44
Set Operation-Complement
O The complement of a set A is a set Ac defined as:
Ac = {x | x ∉ A}
O The complement of a set B with respect to A, also
denoted as A-B, is defined as:
A-B = {x ∈ A | x ∉ B}
Theory of Languages and Automata
Prof. Movaghar
45
Ordered Pairs and n-tuples
O An ordered pair of elements is written
(x,y)
where x is known as the first element, and y is known
as the second element.
O An n-tuple is an ordered sequence of elements
(x1, x2, …, xn)
And is a generalization of an ordered pair.
Theory of Languages and Automata
Prof. Movaghar
46
Ordered Sets and Set Products
O By Cartesian product of two sets A and B, we mean
the set
A×B = {(x,y) | xA , yB}
O Similarly,
A1×A2×…An = {x1A1, x2A2, …, xnAn}
Theory of Languages and Automata
Prof. Movaghar
47
Relations
A relation ρ between sets A and B is a subset of A×B:
ρ  A×B
O The domain of ρ is defined as
Dρ = {x  A | for some y  B, (x,y)  ρ}
O The range of ρ is defined as
Rρ = {y  B | for some x  A, (x,y)  ρ}
O If ρ  A×A, then ρ is called a ”relation on A”.
Theory of Languages and Automata
Prof. Movaghar
48
Types of Relations on Sets
O A relation ρ is reflexive if
(x,x) ρ, for each x A
O A relation ρ is symmetric if, for all x,y A
(x,y)  ρ implies (y,x) ρ
O A relation ρ is antisymmetric if, for all x,yA
(x,y)  ρ and (y,x)  ρ implies x=y
O A relation ρ is transitive if, for all x,y,z A
(x,y)  ρ and (y,z)  ρ implies (x,z)  ρ
Theory of Languages and Automata
Prof. Movaghar
49
Partial Order and Equivalence Relations
O A relation ρ on a set A is called a partial
ordering of A if ρ is reflexive, anti-symmetric,
and transitive.
O A relation ρ on a set A is called an equivalence
relation if ρ is reflexive, symmetric, and
transitive.
Theory of Languages and Automata
Prof. Movaghar
50
Total Ordering
O A relation ρ on a set A is a total ordering if ρ
is a partial ordering and, for each pair of
elements (x,y) in A×A at least one of (x,y)ρ
or (y,x)ρ is true.
Theory of Languages and Automata
Prof. Movaghar
51
Inverse Relation
O For any relation ρ  A×B, the inverse of ρ is
defined by
ρ-1 = {(y,x) | (x,y)  ρ}
O
If Dρ and Rρ are the domain and range of ρ, then
Dρ-1 = Rρ and Rρ-1 = Dρ
Theory of Languages and Automata
Prof. Movaghar
52
Equivalence
O Let ρ  A×A be an equivalence relation on A. The
equivalence class of an element x is defined as
[x] = {y  A | (x,y)  ρ}
O An equivalence relation on a set partitions the set.
Theory of Languages and Automata
Prof. Movaghar
53
Functions
A relation f  A× B is a function if it has the property
(x,y)  f and (x,z)  f implies y = z
O If f  A× B is a function, we write
f: A → B
and say that f maps A into B. We use the common
notation
Y = f(x)
to mean (x,y)  f.
Theory of Languages and Automata
Prof. Movaghar
54
Functions (cont.)
O As before, the domain of f is the set
Df = {x A | for some y B, (x,y)  f}
and the range of f is the set
Rf = {y B | for some x  A, (x,y)  f}
O If Df  A, we say the function is a partial function;
if Df = A, we say that f is a total function.
Theory of Languages and Automata
Prof. Movaghar
55
Functions (cont.)
O If x  Df, we say that f is defined at x; otherwise f
is undefined at x.
O If Rf = B, we say that f maps Df onto B.
O If a function f has the property
f(x) = z and f(y) = z implies x = y
then f is a one-to-one function.
If f: A → B is a one-to-one function, f gives a one-to
one correspondence between elements of its domain
and range.
Theory of Languages and Automata
Prof. Movaghar
56
Functions (cont.)
O Let X be a set and suppose A  X. Define function
CA: X → {0,1}
such that CA(x) = 1, if x A; CA(x) = 0, otherwise.
CA(x) is called the characteristic function of set A
with respect to set X.
O If f  A× B is a function, then the inverse of f is the
set
f--1 = {(y,x) | (x,y)  f}
f--1 is a function if and only if f is one-to-one.
Theory of Languages and Automata
Prof. Movaghar
57
Functions (cont.)
O
Let f: A → B be a function, and suppose that X 
A. Then the set
Y = f(X) = {y  B | y = f(x) for some x  X}
is known as the image of X under f.
O Similarly, the inverse image of a set Y included in
the range of f is
f -1(Y) = {x  A | y = f(x) for some y  Y}
Theory of Languages and Automata
Prof. Movaghar
58
Cardinality
O Two sets A and B are of equal cardinality, written as
|A| = |B|
if and only if there is a one-to-one function f: A → B
that maps A onto B.
O We write
|A| ≤ |B|
if B includes a subset C such that |A| = |C|.
O If |A| ≤ |B| and |A| ≠ |B|, then A has cardinality less
than that of B, and we write
|A| < |B|
Theory of Languages and Automata
Prof. Movaghar
59
Cardinality (cont.)
J = {1, 2, …} and Jn = {1, 2, …, n}.
O A sequence on a set X is a function f: J → X. A sequence
O Let
may be written as
f(1), f(2), f(3), …
However, we often use the simpler notation
x1, x2, x3, …., xi  X
O A finite sequence of length n on X is a function
f: Jn → X, usually written as
x1, x2, x3, …., xn, xi  X
O The sequence of length zero is the function f: → X.
Theory of Languages and Automata
Prof. Movaghar
60
Finite and Infinite Sets
O A set A is finite if |A| = |Jn| for some integer
n ≥ 0, in which case we say that A has
cardinality n.
O A set is infinite if it is not finite.
O A set X is denumerable if |X| = |J|.
O A set is countable if it is either finite or
denumerable.
O A set is uncountable if it is not countable.
Theory of Languages and Automata
Prof. Movaghar
61
Some Properties
O Proposition: Every subset of J is countable.
Consequently, each subset of any denumerable set is
countable.
O Proposition: A function f: J → Y has a countable
range. Hence any function on a countable domain
has a countable range.
O Proposition: The set J × J is denumerable.
Therefore, A × B is countable for arbitrary
countable sets A and B.
Theory of Languages and Automata
Prof. Movaghar
62
Some Properties (cont.)
O Proposition: The set A  B is countable whenever A
and B are countable sets.
O Proposition: Every infinite set X is at least
denumerable ; that is |X| ≥ |J|.
O Proposition: The set of all infinite sequence on
{0, 1}
is uncountable.
Theory of Languages and Automata
Prof. Movaghar
63
Some Properties (cont.)
O Proposition: (Schröder-Bernestein Theorem)
For any set A and B, if |A| ≥ |B| and |B| ≥ |A|, then
|A| = |B|.
O Proposition: (Cantor’s Theorem)
For any set X,
|X| < |P(X)|.
Theory of Languages and Automata
Prof. Movaghar
64
1-1 correspondence Q↔N
O Proof (dove-tailing):
Theory of Languages and Automata
Prof. Movaghar
65
Countable Sets
O Any subset of a countable set
O The set of integers, algebraic/rational numbers
O The union of two/finite number of countable sets
O Cartesian product of a finite number of countable sets
O The set of all finite subsets of N
O Set of binary strings
Theory of Languages and Automata
Prof. Movaghar
66
Diagonal Argument
Theory of Languages and Automata
Prof. Movaghar
67
Uncountable Sets
O R, R2, P(N)
O The intervals [0,1), [0, 1], (0, 1)
O The set of all real numbers
O The set of all functions from N to {0, 1}
O The set of functions N → N
O Any set having an uncountable subset
Theory of Languages and Automata
Prof. Movaghar
68
Transfinite Cardinal Numbers
O Cardinality of a finite set is simply the number of
elements in the set.
O Cardinalities of infinite sets are not natural numbers,
but are special objects called transfinite cardinal
numbers.
O 0:|N|, is the first transfinite cardinal number.
O
continuum hypothesis claims that |R|=1, the second
transfinite cardinal.
Theory of Languages and Automata
Prof. Movaghar
69
Alphabet
O A finite and nonempty set of symbols
(usually shown by ∑ or Г).
O Example:
= {𝑎, 𝑏, 𝑐, … , 𝑧}
Theory of Languages and Automata
Prof. Movaghar
70
Strings
O A finite list of symbols from an alphabet
O Example:
house
O If ω is a string over ∑, the length of ω, written |ω|, is
the number of symbols that it contains.
O The empty string (ε or λ) is the string of length zero.
λω = ωλ = ω
O String z is a substring of ω if z appears consecutively
within ω.
Theory of Languages and Automata
Prof. Movaghar
71
Operations on Strings
O Reverse of a string:
w = a1a2…an
wR = an…a2a1
ababaaabbb
bbbaaababa
O Concatenation:
w = a1a2…an
v = b1b2…bm
wv = a1a2…an b1b2…bm
|wv| = |w| + |v|
Theory of Languages and Automata
abba
bbbaaa
abbabbbaaa
Prof. Movaghar
72
Operations on Alphabet
O *:
 The set of all strings that can be produced from ∑.
O +:
 The set of all strings, excluding λ, that can be produced from ∑.
 Suppose that λ={λ}. Then:
Theory of Languages and Automata
Prof. Movaghar
73
Languages
O A set of strings
O Any language on the alphabet ∑ is a subset of ∑*.
O Examples:
Theory of Languages and Automata
Prof. Movaghar
74
Operations on Languages
Languages are a special kind of sets and operations on sets can be
defined on them as well.
O Union
O Intersection
O Relative Complement
O Complement
λ, b, aa, ab, bb, aaa, …}
Theory of Languages and Automata
Prof. Movaghar
75
Operations on Languages (cont.)
O Reverse of a language
O Example:
Theory of Languages and Automata
Prof. Movaghar
76
Operations on Languages (cont.)
O Concatenation
O Example:
Theory of Languages and Automata
Prof. Movaghar
77
Operations on Languages (cont.)
O *:
Kleene *
 The set of all strings that can be produced by concatenation
of strings of a language.
 Example:
O +:
 The set of all strings, excluding λ, that can be produced by
concatenation of strings of a language.
Theory of Languages and Automata
Prof. Movaghar
78
Gödel Numbering
O Let ∑ be an alphabet containing
n objects. Let h: ∑ → Jn
be an arbitrary one-to-one correspondence. Define
function f as:
f: ∑* → N
such that f(ε) =0; f(w.v) = n f(w) + h(v), for w ∑* and v ∑.
f is called a Gödel Numbering of ∑*.
O Proposition:
∑* is denumerable.
O Proposition: Any language on an alphabet is countable.
Theory of Languages and Automata
Prof. Movaghar
79