CS621/CS449
Artificial Intelligence
Lecture Notes
Set 6: 22/09/2004, 24/09/2004,
1/10/2004, 6/10/2004
Instructor: Prof. Pushpak Bhattacharyya
13/08/2004
CS-621/CS-449 Lecture Notes
Contents
• Complexity of Feedforward Network (FFN)
training
• Positive Linear Confinement (PLC) problem
• Computational complexity basics
• PLC is NP-Complete
22/09/2004 - 06/10/2004
CS-621/CS-449 Lecture Notes
Prof. Pushpak Bhattacharyya
IIT Bombay
Complexity of Feedforward Network
(FFN) Training
• Observation
– Almost all problems that use FFN with BP
require a very large amount of time.
• Rivest and Blum (1992)
– Training of 3-layer FFN is NP-complete
22/09/2004 - 06/10/2004
CS-621/CS-449 Lecture Notes
Prof. Pushpak Bhattacharyya
IIT Bombay
Decision Problem for FFN Training
• Given a set of labeled patterns, does there exist
two hyperplanes such that patterns of only one
kind are confined to a quadrant?
In other words,
• Can a 3-node NN be trained on the given
positive and negative examples?
• Does there exist two hyperplanes defined by h0
and h1 which confine ALL and ONLY points of
one kind? (Linear Confinement Problem)
22/09/2004 - 06/10/2004
CS-621/CS-449 Lecture Notes
Prof. Pushpak Bhattacharyya
IIT Bombay
Positive Linear Confinement(PLC)
problem
• The PLC decision problem is computationally
hard i.e. as the number of points grows, the time
taken to find the planes grows exponentially.
• Equivalent statement:
– Training of a 3-node NN will take time
22/09/2004 - 06/10/2004
CS-621/CS-449 Lecture Notes
Prof. Pushpak Bhattacharyya
IIT Bombay
Computational Complexity Basics
TURING MACHINE
Finite state head
Infinite tape
• Time Complexity
– The number of steps taken by a Turing
Machine (TM) to complete the computation as
a function of the length of the input
22/09/2004 - 06/10/2004
CS-621/CS-449 Lecture Notes
Prof. Pushpak Bhattacharyya
IIT Bombay
Computational Complexity Basics
• Space Complexity
– The number of cells on the tape needed to
complete the computation, as a function of the
length of the input
• Kolmogorov Complexity
– The length of the shortest program (as a
function of input length) to express the
computation
22/09/2004 - 06/10/2004
CS-621/CS-449 Lecture Notes
Prof. Pushpak Bhattacharyya
IIT Bombay
Complexity Expressions for AI Problems
• Most problems in AI are pattern recognition
problems
• AI problems involve enumerating large number
of cases
• Numerical problems are described in terms of
space and time complexity, whereas, AI
problems are described using Kolmogorov
complexity
22/09/2004 - 06/10/2004
CS-621/CS-449 Lecture Notes
Prof. Pushpak Bhattacharyya
IIT Bombay
Complexity Classes
• Class P
– It is the class of decision problems which can
be computed in polynomial time by a
deterministic TM
• Class NP
– It is the class of decision problems which can
be computed in polynomial time by a nondeterministic TM
– For this class of problems, once the solution
is given, it is easy to verify it
22/09/2004 - 06/10/2004
CS-621/CS-449 Lecture Notes
Prof. Pushpak Bhattacharyya
IIT Bombay
Complexity Classes
• Since Deterministic TMs are a subset of Nondeterministic TMs,
P  NP
• Complexity Class:
– This is a class of problems which share the
complexity measure
– There exists “transformations” of lesser
complexity among the problems
22/09/2004 - 06/10/2004
CS-621/CS-449 Lecture Notes
Prof. Pushpak Bhattacharyya
IIT Bombay
Complexity Classes
• Completeness within a complexity class
– If one of the problems within the class is
solved in less time than indicated by the
complexity, all of them can be solved in that
amount of ‘less’ time.
– This happens because of the existence of
transformations of less complexity among the
problems
22/09/2004 - 06/10/2004
CS-621/CS-449 Lecture Notes
Prof. Pushpak Bhattacharyya
IIT Bombay
Transformation Procedure
• A known Complexity problem is chosen
•
Arbitrary instance of
“known complexity”
problem
Transformation
procedure
Constructed instance of
problem whose
complexity is desired
– The transformation procedure should take
less time than the inherent complexity of the
known complexity problem
22/09/2004 - 06/10/2004
CS-621/CS-449 Lecture Notes
Prof. Pushpak Bhattacharyya
IIT Bombay
Set Splitting Problem
• Here, we introduce Set Splitting problem which
is later used in proving that PLC is NP-Complete
• Given a set S= {s1,s2,s3,…sn} and subsets Ci of S
such that  c  S , can S be split into two
subsets S1 and S2 such that
C  S ,C  S
S1  S2  S and
i
i
i
22/09/2004 - 06/10/2004
1
i
2
CS-621/CS-449 Lecture Notes
Prof. Pushpak Bhattacharyya
IIT Bombay
PLC is NP-Complete
1. PLC belongs to NP
• Given two hyperplanes, one can check
easily (in polynomial time) if they confine all
and only the positive points
2. Transformation: Transformation of ‘Set
Splitting’ problem to PLC in polynomial time
• Corresponding to n elements in S, set up a
coordinate system with n axes
xn
•
S = {s1,s2,s3,…sn}
Xn-1 Xn-2 Xn-3
22/09/2004 - 06/10/2004
CS-621/CS-449 Lecture Notes
X1
Prof. Pushpak Bhattacharyya
IIT Bombay
PLC is NP-Complete
•
Transformation of SS problem instance to an
instance of PLC:
1. Mark origin as positive
2. Mark unit-distance points on the axes as
negative
Ex: S={s1,s2,s3}
C1={s1, s3}
C2={s2}
22/09/2004 - 06/10/2004
(0,0,1)
(0,1,0)
+
CS-621/CS-449 Lecture Notes
(0,0,0)
(1,0,0)
Prof. Pushpak Bhattacharyya
IIT Bombay
PLC is NP-Complete
• For each ci, mark
<j1,j2,j3,…,jn> as positive,
where
jk  1 if s k  ci 
 0otherwise 
+
(0,1,1)
(0,0,1)
(0,1,0)
22/09/2004 - 06/10/2004
+
CS-621/CS-449 Lecture Notes
(0,0,0)
(1,0,0)
+
(1,1,0)
Prof. Pushpak Bhattacharyya
IIT Bombay
PLC is NP-Complete
• The transformation implies
– If part: If the set S can be split, two
hyperplanes can be found which can confine
all and only positive points
– Then part: If two hyperplanes can confine ALL
and only positive points, S can be split
22/09/2004 - 06/10/2004
CS-621/CS-449 Lecture Notes
Prof. Pushpak Bhattacharyya
IIT Bombay
Example for SS to PLC transformation
• For the previous set-splitting example, one
possible solution for the corresponding instance
of PLC is
P1: -x1+3x2-x3 = -1/2
P2: 3x1-x2+3x3=-1/2 (see that all positive points
are on the same side as origin)
22/09/2004 - 06/10/2004
CS-621/CS-449 Lecture Notes
Prof. Pushpak Bhattacharyya
IIT Bombay