Local Search (part 1)
Haim Kaplan and Uri Zwick
Algorithms in Action
Tel Aviv University
Last updated: March 2017
1
An example
2
Max Cut
Partition 𝐺 = (𝑉, 𝐸) into 2 parts 𝑆, 𝑉 ∖ 𝑆 such that the
number of edges 𝑢, 𝑣 , 𝑢 ∈ 𝑆, 𝑣 ∈ 𝑉 ∖ 𝑆 is maximized
𝑆
𝑆 =𝑉∖𝑆
3
Few comments
• This problem is NP-hard, best
approximation guarantee is 0.878, NP-hard
to get more than 0.91
• The analogous problem of finding a
minimum cut is easier (how ?)
4
The idea
Start with some cut and
try to improve it by making local changes
Stop when there is no local change that improves the cut
5
Local change
𝑆
𝑆
6
Local change
𝑆
𝑆
7
Local change
𝑆
𝑆
8
Local change
𝑆
𝑆
9
Local change
𝑆
𝑆
10
Local change
𝑆
𝑆
11
Local vs. global OPT
• When we stop we have a cut that we cannot
improve by a local change (local opt)
• This need not be the largest cut (global opt)
12
Local OPT
𝑆
𝑆
A
D
B
C
13
Global OPT
𝑆
𝑆
B
D
A
C
14
Analysis
There are two main parameters of interest:
• Time: How long it takes to get to a local
optimum ?
• Quality of the local optimum ?
15
Quality of local opt
𝑆 =𝑉∖𝑆
𝑆
v
Can this be the situation at a local optimum ?
16
Quality of local opt
𝑆 =𝑉∖𝑆
𝑆
v
∀𝑣 𝛾 𝑣 ≤ 𝛿(𝑣)
𝛿 𝑣 = #edges incident to v that cross the cut
𝛾 𝑣 = #edges incident to v that do not cross the cut
17
Quality of local opt
𝑆=𝑉∖𝑆
𝑆
v
𝛾 𝑣 ≤
𝑣
2
𝑢, 𝑣
𝑢, 𝑣 ∈ 𝑆 𝑜𝑟 𝑢, 𝑣 ∈ 𝑆
𝛿 𝑣
𝑣
≤2
𝑢, 𝑣
𝑢 ∈ 𝑆, 𝑣 ∈ 𝑆
18
Quality of local opt
𝑆=𝑉∖𝑆
𝑆
v
𝛾 𝑣 ≤
𝑣
𝑢, 𝑣
𝑣
𝑢, 𝑣 ∈ 𝑆 𝑜𝑟 𝑢, 𝑣 ∈ 𝑆
𝑂𝑃𝑇 |𝐸|
≤
≤
2
2
𝛿 𝑣
𝑢, 𝑣
≤
𝑢, 𝑣
𝑢 ∈ 𝑆, 𝑣 ∈ 𝑆
𝑢 ∈ 𝑆, 𝑣 ∈ 𝑆
19
Steps to local opt
𝑂( 𝐸 )
Each step the size of the cut increases by at least one
20
Weighted graphs
Partition 𝐺 = (𝑉, 𝐸) into 2 parts 𝑆, 𝑉 ∖ 𝑆 such that the
∑𝒘 𝑢, 𝑣 , 𝑢 ∈ 𝑉, 𝑣 ∈ 𝑉 ∖ 𝑆 is maximized
𝑆
𝑆 =𝑉∖𝑆
21
Weighted graphs
Works the same:
Make a swap if it increases the weight of the cut
𝑂𝑃𝑇 𝑊
≤
≤ 𝑤 𝑢, 𝑣 𝑢 ∈ 𝑆, 𝑣 ∈ 𝑆
2
2
𝑊 = ∑𝑤(𝑢, 𝑣)
22
Hopfield networks
A neural network that is used as memory (Content
Addressable Memory)
Initialized with a noisy or incomplete pattern it converges to
the correct pattern
23
Memory cells (pixels) are nodes
in a weighted undirected graph
𝑠𝑖 ∈ {−1,1}
𝑠𝑖
𝑤𝑖𝑗
𝑠𝑗
24
Retrieving form memory
Initialize with a
tentative content
1
1
−1
-1
1
-1
1
25
Retrieving form memory
Each neuron 𝑖 in
turn checks if
∑𝑗 𝑤𝑖𝑗 𝑠𝑖 𝑠𝑗 > 0 ?
If so then 𝑠𝑖 flips
When no neuron
wants to flip we
found the content
1
1
−1
-1
1
-1
1
26
Retrieving form memory
Each neuron 𝑖 in
turn checks if
𝐴−𝐵 >0?
−1
-1
1
1
If so then 𝑠𝑖 flips
1
When a neuron flips
the value of the cut
increases  converges
to a local max cut
-1
1
27
Storing in memory
We want to
compute weights
such that the local
max cuts have a
specific form
Large (positive) weight
between cells in
different states, small
(negative) weight
between cells of the
same state
−1
-1
1
1
1
-1
1
28
Hebbian learning
Suppose we want to store 𝑥 = (𝑥1 , 𝑥2 , … , 𝑥𝑛 )
Set 𝑤𝑖𝑗 = −𝑥𝑖 𝑥𝑗
29
Hebbian learning
1
1
−1
-1
1
−1
−1
1
-1
1
30
Storing in memory
The desired cut is
even the global max
1
−1
-1
1
1
−1
1
-1
1
31
Hebbian learning
Suppose we want 𝑥 = (𝑥1 , 𝑥2 , … , 𝑥𝑛 ), 𝑦 = 𝑦1 , 𝑦2 , … , 𝑦𝑛 , and
𝑧 = (𝑧1 , 𝑧2 , … , 𝑧𝑛 ) to be local maxima
Set 𝑤𝑖𝑗 = −𝑥𝑖 𝑥𝑗 −𝑦𝑖 𝑦𝑗 −𝑧𝑖 𝑧𝑗
Each neuron 𝑖 in turn checks if ∑𝑗 𝑤𝑖𝑗 𝑠𝑖 𝑠𝑗 > 0 ?
Each neuron 𝑖 in turn checks if
∑𝑗(−𝑥𝑖 𝑥𝑗 −𝑦𝑖 𝑦𝑗 −𝑧𝑖 𝑧𝑗 )𝑠𝑖 𝑠𝑗 > 0 ?
𝑥 is stable iff for all 𝑖
(−𝑥𝑖 𝑥𝑗 −𝑦𝑖 𝑦𝑗 −𝑧𝑖 𝑧𝑗 )𝑥𝑖 𝑥𝑗 ≤ 0 ?
𝑗
32
Hebbian learning
𝑥 is stable iff for all 𝑖
(−𝑥𝑖 𝑥𝑗 −𝑦𝑖 𝑦𝑗 −𝑧𝑖 𝑧𝑗 )𝑥𝑖 𝑥𝑗 ≤ 0
𝑗
−𝑥𝑖2
𝑥
2
− 𝑦𝑖 𝑥𝑖 𝑥, 𝑦 − 𝑧𝑖 𝑥𝑖 𝑥, 𝑧 ≤ 0
−𝑛 − 𝑦𝑖 𝑥𝑖 𝑥, 𝑦 − 𝑧𝑖 𝑥𝑖 𝑥, 𝑧 ≤ 0
−𝑦𝑖 𝑥𝑖 𝑥, 𝑦 − 𝑧𝑖 𝑥𝑖 𝑥, 𝑧 ≤ 𝑛
This will hold if we are not trying to store too many vectors or
if the vectors are nearly orthogonal…
33
Storage capacity:
Stochastic analysis
How may random −1/1 vectors can we store with high
probability ?
≤ 0.138𝑛
34
Back to max cut
35
Steps to local opt
We construct a example with a long running time
The example uses two kinds of vertices
36
Steps to local opt
𝑒2
𝑒3 𝑣
𝑒4
𝑒1
𝑤 𝑒3 > 𝑤 𝑒1 + 𝑤 𝑒2 + 𝑤 𝑒4
37
Steps to local opt
𝑒2
𝑒3 𝑣
𝑒4
𝑒1
𝑤 𝑒3 > 𝑤 𝑒1 + 𝑤 𝑒2 + 𝑤 𝑒4
38
Steps to local opt
𝑒2
𝑒3 𝑣
𝑒4
𝑒1
𝑤 𝑒3 > 𝑤 𝑒1 + 𝑤 𝑒2 + 𝑤 𝑒4
39
Steps to local opt
𝑒2
𝑒3 𝑣
𝑒4
𝑒3
𝑒1
𝑤 𝑒4 > 𝑤 𝑒2 , 𝑤 𝑒3 > 𝑤 𝑒1
𝑒2
𝑣
𝑒4
𝑤 𝑒3 > 𝑤 𝑒1 + 𝑤 𝑒2 + 𝑤 𝑒4
𝑒1
𝑤 𝑒2 + 𝑤 𝑒3 > 𝑤 𝑒1 + 𝑤 𝑒4
40
Steps to local opt
𝑒2
𝑒3 𝑣
𝑒4
𝑒3
𝑒1
𝑤 𝑒4 > 𝑤 𝑒2 , 𝑤 𝑒3 > 𝑤 𝑒1
𝑒2
𝑣
𝑒4
𝑤 𝑒3 > 𝑤 𝑒1 + 𝑤 𝑒2 + 𝑤 𝑒4
𝑒1
𝑤 𝑒2 + 𝑤 𝑒3 > 𝑤 𝑒1 + 𝑤 𝑒4
41
Steps to local opt
𝑒2
𝑒3 𝑣
𝑒4
𝑒3
𝑒1
𝑤 𝑒4 > 𝑤 𝑒2 , 𝑤 𝑒3 > 𝑤 𝑒1
𝑒2
𝑣
𝑒4
𝑤 𝑒3 > 𝑤 𝑒1 + 𝑤 𝑒2 + 𝑤 𝑒4
𝑒1
𝑤 𝑒2 + 𝑤 𝑒3 > 𝑤 𝑒1 + 𝑤 𝑒4
42
Steps to local opt
𝑒2
𝑒3 𝑣
𝑒4
𝑒3
𝑒1
𝑤 𝑒4 > 𝑤 𝑒2 , 𝑤 𝑒3 > 𝑤 𝑒1
𝑒2
𝑣
𝑒4
𝑤 𝑒3 > 𝑤 𝑒1 + 𝑤 𝑒2 + 𝑤 𝑒4
𝑒1
𝑤 𝑒2 + 𝑤 𝑒3 > 𝑤 𝑒1 + 𝑤 𝑒4
43
Steps to local opt
𝑒2
𝑒3 𝑣
𝑒4
𝑒3
𝑒1
𝑤 𝑒4 > 𝑤 𝑒2 , 𝑤 𝑒3 > 𝑤 𝑒1
𝑒2
𝑣
𝑒4
𝑤 𝑒3 > 𝑤 𝑒1 + 𝑤 𝑒2 + 𝑤 𝑒4
𝑒1
𝑤 𝑒2 + 𝑤 𝑒3 > 𝑤 𝑒1 + 𝑤 𝑒4
44
Steps to local opt
45
Steps to local opt
46
Steps to local opt
47
Steps to local opt
48
Steps to local opt
49
Steps to local opt
50
Steps to local opt
51
Steps to local opt
52
Steps to local opt
53
Steps to local opt
54
Steps to local opt
55
Steps to local opt
56
Steps to local opt
57
Steps to local opt
58
Steps to local opt
59
Steps to local opt
𝑣0
𝑣1
Moral: 2 flips of 𝑣0 caused 4 flips of 𝑣1
60
Steps to local opt
𝑣0
𝑣1
𝑣2
61
Steps to local opt
𝑣0
𝑣1
𝑣2
62
Steps to local opt
𝑣0
𝑣1
𝑣2
63
Steps to local opt
𝑣0
𝑣1
𝑣2
64
Steps to local opt
𝑣0
𝑣1
𝑣2
65
Steps to local opt
𝑣0
𝑣1
𝑣2
66
Steps to local opt
𝑣0
𝑣1
𝑣2
67
Steps to local opt
𝑣0
𝑣1
𝑣2
68
Steps to local opt
𝑣0
𝑣1
𝑣2
69
Steps to local opt
𝑣0
𝑣1
𝑣2
70
Steps to local opt
𝑣0
𝑣1
𝑣2
71
Steps to local opt
𝑣0
𝑣1
𝑣2
72
Steps to local opt
𝑣0
𝑣1
𝑣2
73
Steps to local opt
𝑣0
𝑣1
𝑣2
At this point 𝑣2 flipped 4 times and the circled gadget is in its
initial configuration
74
Steps to local opt
𝑣0
𝑣1
𝑣2
75
Steps to local opt
𝑣0
𝑣1
𝑣2
76
Steps to local opt
𝑣0
𝑣1
𝑣2
77
Steps to local opt
𝑣0
𝑣1
𝑣2
78
Steps to local opt
𝑣0
𝑣1
𝑣2
79
Steps to local opt
𝑣0
𝑣1
𝑣2
80
Steps to local opt
𝑣0
𝑣1
𝑣2
81
Steps to local opt
𝑣0
𝑣1
𝑣2
82
Steps to local opt
𝑣0
𝑣1
𝑣2
83
Steps to local opt
𝑣0
𝑣1
𝑣2
84
Steps to local opt
𝑣0
𝑣1
𝑣2
85
Steps to local opt
𝑣0
𝑣1
𝑣2
86
Steps to local opt
𝑣0
𝑣1
𝑣2
By induction we can make 𝑣𝑘 flip 2𝑘+1 times
87
#Steps - Summary
• Still 𝑂 𝑊 , 𝑊 = ∑𝑒∈𝐸 𝑤 𝑒 , if all weights
are integers (pseudo polynomial)
• We can prevent long running time if we
only do a move if it improves by at least
2𝜖𝑤(𝑆,𝑆)
𝑛
• Then the # of moves is bounded by
𝑛
𝑂 log(𝑊)
𝜖
88
Big improvement - analysis
𝑆
𝑆
v
∀𝑣 𝑤 Γ 𝑣
≤𝑤 Δ 𝑣
2𝜖𝑤(𝑆, 𝑆)
+
𝑛
Δ 𝑣 = set of edges incident to v that cross the cut
Γ 𝑣 = set of edges incident to v that do not cross the cut
Big improvement - analysis
𝑆=𝑉∖𝑆
𝑆
v
𝑤(Γ 𝑣 ) ≤
𝑣
𝑤 Δ 𝑣
𝑣
2𝜖𝑤(𝑆, 𝑆)
+
𝑛
2𝑤 𝑢, 𝑣 𝑢, 𝑣 ∈ 𝑆 𝑜𝑟 𝑢, 𝑣 ∈ 𝑆
≤ 2𝑤 𝑢, 𝑣 𝑢 ∈ 𝑆, 𝑣 ∈ 𝑆 + 2𝜖𝐿
90
Big improvement - analysis
𝑤 𝑢, 𝑣 𝑢, 𝑣 ∈ 𝑆 𝑜𝑟 𝑢, 𝑣 ∈ 𝑆
≤ 𝑤 𝑢, 𝑣 𝑢 ∈ 𝑆, 𝑣 ∈ 𝑆 + 𝜖𝐿
𝑊 − 𝐿 ≤ 𝐿 + 𝜖𝐿
𝑊 ≤ 2+𝜖 𝐿
1
1
𝑂𝑃𝑇 ≤
𝑊≤𝐿
2+𝜖
2+𝜖
Max-cut notes
• The ½ approximation guarantee is not that
impressive: A random partition gives this
guarantee in expectation
• Can also get ½ approximation
deterministically by inserting the vertices
one by one to the ``right’’ side of the cut
92
Max-cut notes
• But still, our local search scheme gives in
practice larger cuts.
93