Cooperative Heuristic Search with
Software Agents
Antti Halme
[email protected]
April 24, 2014
Outline
Cooperation and heuristic search
The A! algorithm
Solving n-puzzles with A!
Conclusion
Cooperative Heuristic Search with Software Agents
Antti Halme
2/30
Outline
Cooperation and heuristic search
The A! algorithm
Solving n-puzzles with A!
Conclusion
Cooperative Heuristic Search with Software Agents
Antti Halme
3/30
Cooperation
co·op·er·ate
I
to work together
I
to work with another person or group to do something
I
to be helpful by doing what someone asks or tells you to do
I
to act in a way that makes something possible or likely
I
to produce the right conditions for something to happen
(Merriam-Webster)
Cooperative Heuristic Search with Software Agents
Antti Halme
3/30
Heuristic Search
I
Searching is a fundamental computing task
I
A heuristic provides focus to searching efforts
I
The most well-known heuristic search algorithm is the A∗
Uninformed search
Heuristic search
Cooperative Heuristic Search with Software Agents
Antti Halme
4/30
A∗ search1
I
I
I
Informed best-first graph search algorithm
Single-source shortest path problem (SSSP)
Fringe nodes ranked by a cost estimate
f (n)
|{z}
=
cost estimate
I
I
g(n)
|{z}
known distance
+
h(n)
|{z}
estimated remaining
Admissible heuristics never overestimate, h(n) ≤ h∗ (n)
Consistent heuristics guarantee optimality
h(n) ≤ d(n, n0 ) + h(n0 )
I
Optimally efficient on given heuristic
I
No algorithm can expand fewer nodes, except in
tie-breaks
1
Peter Hart, Nils Nilsson, and Bertram Raphael. A Formal Basis for the Heuristic Determination of Minimum
Cost Paths. IEEE Transactions on Systems Science and Cybernetics, 4(2):100–107, 1968.
Cooperative Heuristic Search with Software Agents
Antti Halme
5/30
Constructing cooperative search
I
The goal
I
I
The hypothesis
I
I
I
Cooperation is communication
Agents share and make good use of progress information
The method
I
I
Cooperating search agents outperform agents in isolation
The idea
I
I
Search faster as a collective, leverage parallel hardware
A secondary ranking heuristic, a dynamic tiebreaker
The mechanism
I
Asynchronous messaging, natural concurrency,
implicit randomness, nondeterministic exploration
Unus pro omnibus, omnes pro uno.
Cooperative Heuristic Search with Software Agents
Antti Halme
6/30
Outline
Cooperation and heuristic search
The A! algorithm
Solving n-puzzles with A!
Conclusion
Cooperative Heuristic Search with Software Agents
Antti Halme
7/30
Overview of A!
I
A∗ + cooperation + concurrency = A! (a-bang)
I
A cooperative heuristic search algorithm for the SSSP
I
Search agents run an upgraded version of A∗
I
Shared information included as a secondary heuristic, ĥ
I
Simplest case: share best encountered (n, h(n))-pair
I
Vanilla A∗ maintains a node priority queue sorted by
f (n) = g(n) + h(n),
A! takes k equal-valued nodes from the top, ranks by ĥ
Cooperative Heuristic Search with Software Agents
Antti Halme
7/30
Overview of A!
t
s
G
A
u
v
w
B
A! search on a grid graph with two agents, A and B. Between nodes t and u,
of equal distance to G (1st heuristic), agent A chooses u, because its closer
(2nd heuristic) to best node marked with star, discovered by B.
Cooperative Heuristic Search with Software Agents
Antti Halme
8/30
Optimality of A!
I
I
Proof sketch based on an argument for A∗ itself2
Cost estimates, f -values, are nondecreasing for all paths
I
I
Optimal path to a node is found before the node is opened
I
I
Consistent heuristic
Expansion from the fringe, edge connection
Let S be a subset of all equally good candidate nodes, ES
all explored nodes with a successor in S
1. f (n0 ) is the same for all n0 ∈ S, so for n ∈ ES
f (n0 ) = g(n0 ) + h(n0 ) = g(n) + d(n, n0 ) + h(n0 ) ≥ g(n) + h(n) = f (n)
2. For a node to be opened before an optimal path there is
found, another fringe node m with a better f -value is
implied, but then f (n) < f (m) and f (n) > f (m).
2
Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach, 3rd ed. Prentice Hall Press, 2009.
Cooperative Heuristic Search with Software Agents
Antti Halme
9/30
A! cooperation architecture
I
Agents process async messages at their own pace
I
Concurrency, nondeterminism; implicit randomness
I
Best-effort notion of a globally superior reference node
I
A message broker entity can facilitate communication
I
I
I
Short-circuiting best-update leads to a momentum effect
I
I
I
Instantaneous diffusion is not the goal
Publish/subscribe-topology (cf. web chat)
Prefer nodes near the previously explored ones
Manifest already with a single agent
Actor abstraction, fast termination, diversification, . . .
Cooperative Heuristic Search with Software Agents
Antti Halme
10/30
A! details – A!Search, A!Solver, A!Select
Algorithm 1 : A!Search
Require: N > 0, NODE start, PREDICATE isGoal, HEURISTIC h, HEURISTIC ĥ
Ensure: path from start to nearest node satisfying isGoal is shortest possible
mb ← MsgBroker ()
for i = 0 to N do
workers[i] ← A!Solver (mb.portOut, mb.portIn, s, isGoal, h, ĥ)
end for
for each worker in workers in parallel do
worker .launch()
end for
wait for termination
return path ← getPath(workers)
Cooperative Heuristic Search with Software Agents
Antti Halme
11/30
Algorithm 2 : A!Solver
Require: PORT portIn, portOut, NODE start, PRED isGoal, HEUR h, ĥ
1: openHeap ← FibonacciHeaphINTEGER, NODEi
2: closedSet ← SethNODEi
3: pathMap ← MaphNODE, NODEi
4: current ← start
5: repeat
6:
if isGoal(current) then terminate(current, start, pathMap) end if
7:
closedSet.add(current)
8:
for each n in current.getNeighbors() do
9:
if closedSet.contains(n) then continue end if
10:
g ← current.g + dist(current, n)
11:
f ← g + h(n)
12:
improved ← openHeap.update(n, f )
13:
if improved then pathMap.update(n, current) end if
14:
end for
15:
peekList ← openHeap.getPeekList()
16:
if isEmpty (peekList) then terminate() end if
17:
current ← A!Select(peekList, portIn, portOut, h, ĥ)
18:
openHeap.remove(current)
19: until termination
Cooperative Heuristic Search with Software Agents
Antti Halme
12/30
Algorithm 3 : A!Select
Require: LIST peekList, PORT portIn, portOut, HEURISTIC h, ĥ
Ensure: select is the most promising node in peekList according to ĥ on best
1: update, updateH ← asyncRecv (portIn)
2: if updateH < bestH then
3:
best, bestH ← update, updateH
4: end if
5: select ← peekList.pop()
6: selectD ← ĥ(select, best)
7: for each node in peekList do
8:
d ← ĥ(node, best)
9:
if d < selectD then select, selectD ← node, d end if
10: end for
11: if h(select) < bestH then
12:
best, bestH ← select, selectH
13:
asyncSend(portOut, {best, bestH})
14: end if
15: return select
Cooperative Heuristic Search with Software Agents
Antti Halme
13/30
A! tradeoffs and challenges
I
Shared memory vs. distributed
I
Cooperation details: what to share and how, usage
I
Maintaining diversity: partitioning, hashing, . . .
I
IDA∗ and maintaining memory-efficiency
I
Clean, fast termination
I
Priority queue bottleneck
I
Agent abstraction: actors, coroutines, . . .
Cooperative Heuristic Search with Software Agents
Antti Halme
14/30
Outline
Cooperation and heuristic search
The A! algorithm
Solving n-puzzles with A!
Conclusion
Cooperative Heuristic Search with Software Agents
Antti Halme
15/30
The n-puzzle
8 6 7
2 5 4
1
3
1 2 3
4 5 6
7 8
A start state
A standard goal state
Cooperative Heuristic Search with Software Agents
Antti Halme
15/30
The n-puzzle
I
Classic sliding tile puzzle with a long history3
I
Turn the start state into the target state by sliding tiles
I
Literally a toy problem
I
I
8-puzzle avg branching factor ∼3, avg solution 22 steps
I
I
I
322 tree states, 9!
2 ≈ 180k graph
15-puzzle 1.3 × 1012 , 24-puzzle ∼1025
Hardest 8-p on 31 steps, 15-p 80, 24-p 152–208
I
3
Finding k -bound sequence for general m × m is NP-C4
cf. God’s Number for the Rubik’s Cube is 20 (Rokicki et al., 2010)
Jerry Slocum and Dic Sonneveld. The 15 Puzzle Book. The Slocum Puzzle Foundation, 2006.
4
Daniel Ratner and Manfred Warmuth. Finding a Shortest Solution for the NxN Extension of the 15-PUZZLE Is
Intractable. AAAI ’86, pages 168–172, 1986.
Cooperative Heuristic Search with Software Agents
Antti Halme
16/30
Heuristics for n-puzzle
I
I
Misplaced tile count
Manhattan distance
I
I
The sum of distances the tiles are from their target
positions, counted as moves along the grid
Manhattan distance with linear collisions
I
Two tiles on the right row, but in the wrong order must pass
each other to reach their targets
I
Walking distance
I
Pattern databases
Additive/disjoint pattern databases
I
I
I
Store precomputed solutions to sub-problems; disjoint sets
ensure combination heuristic remains admissible
Learned heuristics
Cooperative Heuristic Search with Software Agents
Antti Halme
17/30
Computational setting
I
I
Randomly generated 8- and 15-puzzle instances
Grouped by optimal path length
I
I
I
Substantial variance within each group
Simple A! implementation based on A∗ by Brian Borowski
Focus on nodes opened by winning agent
I
Correlates with runtime, total opened count
I
Comparison with A∗ and non-coop randomized A∗ , A?
I A!Select → A*Select, A?Select
I
Test execution on Aalto SCI Science-IT project resources
I
Triton cluster of mixed multi-core blade servers
I 2.6GHz Opteron 2435, 2.67GHz Xeon X5650, and 2.8GHz Xeon E5 2680 v2
Cooperative Heuristic Search with Software Agents
Antti Halme
18/30
Results: Cooperation benefit and scalability
15−puzzle runs, length groups 40−59, 1−8 agents, PDB heuristic.
Visited vertices for winning agent, cooperative A! search
900000
1 agent
2 agents
4 agents
6 agents
8 agents
Par
1 A lin. reg.
2 A lin. reg.
4 A lin. reg.
6 A lin. reg.
8 A lin. reg.
800000
700000
600000
500000
400000
300000
200000
100000
0
0
100000
200000
300000
400000
500000
600000
Visited vertices for winning agent, vanilla A* search
700000
800000
900000
Relative performance of A∗ (x-axis) and A! (y-axis). The black line is par, so
data points below it represent instances for which A! performs better than A∗ .
Agent count is evaluated in five batches – 1, 2, 4, 6, and 8 agents – with the
respective trend lines showing how the methods compare.
Cooperative Heuristic Search with Software Agents
Antti Halme
19/30
Results: Cooperation benefit and scalability
15−puzzle runs, length groups 40−59, 1−8 agents, PDB heuristic.
Visited vertices for winning agent, cooperative A! search
900000
1 agent
2 agents
4 agents
6 agents
8 agents
Par
1 A lin. reg.
2 A lin. reg.
4 A lin. reg.
6 A lin. reg.
8 A lin. reg.
800000
700000
600000
500000
400000
300000
200000
100000
0
0
100000
200000
300000
400000
500000
600000
Visited vertices for winning agent, random A? search
700000
800000
900000
Relative performance of A? and A!. As before, the data points and trend lines
below par-line reflect the benefit from cooperation. The slopes vary from
7
8
around 10
to 10
, reflecting a 25 − 40% performance difference in favor of A!.
Cooperative Heuristic Search with Software Agents
Antti Halme
20/30
Results: Cooperation benefit and scalability
15−puzzle runs, length groups 40−59, 1−8 agents, PDB heuristic.
350000
300000
250000
A*, 1−8 agents
A?, 1 agent
A!, 1 agent
A?, 2 agents
A!, 2 agents
A?, 4 agents
A!, 4 agents
A?, 6 agents
A!, 6 agents
A?, 8 agents
A!, 8 agents
200000
150000
100000
50000
0
A*
G40 A?
A!
A*
G41 A?
A!
A*
G42 A?
A!
A*
G43 A?
A!
A*
G44 A?
A!
A*
G45 A?
A!
A*
G46 A?
A!
A*
G47 A?
A!
A*
G48 A?
A!
A*
G49 A?
A!
A*
G50 A?
A!
A*
G51 A?
A!
A*
G52 A?
A!
A*
G53 A?
A!
A*
G54 A?
A!
A*
G55 A?
A!
A*
G56 A?
A!
A*
G57 A?
A!
A*
G58 A?
A!
A*
G59 A?
A!
Visited vertices for winning agent, group median
400000
General A∗ , A? and A! performance trends. 100 inst./group. 1, 2, 4, 6, and 8
agents. Cooperation appears to be more beneficial with harder instances.
Cooperative Heuristic Search with Software Agents
Antti Halme
21/30
15−puzzle runs, length groups 40−59, 1−8 agents, PDB heuristic.
2
A*
A? normalized group mean, 1−−8A
A! normalized group mean, 1−−8A
A? mean of norm. group means, 1−−8A
A! mean of norm. group means, 1−−8A
46
Visited vertices for winning agent, means by group, normalized to A*
52
59
5851
44 41
48 55
53
56
1.5
57
49
5042
54 40
45
47 43
I A∗ -normalized length groups
1
I Scaling benefit from more agents
52
46
44
5851
5941
55
5740
50
4843
45
5649
5347
42
A*
I Overlapping trend lines
I
Rudimentary asymptotics
54
0.5
0
A*
1A? 2A?
4A?
6A?
8A?
1A! 2A!
4A!
6A!
8A!
Cooperative Heuristic Search with Software Agents
Antti Halme
22/30
Results: Heuristic impact
15−puzzle runs, lengths 30−57, 1−8 agents, LC/PDB heuristic.
Visited vertices, 6−6−3 PDB heuristic
300000
1 agents
2 agents
4 agents
6 agents
8 agents
1 agent lin. reg.
2 agents lin. reg.
4 agents lin. reg.
6 agents lin. reg.
8 agents lin. reg.
250000
200000
150000
100000
50000
0
0
100000
200000
300000
400000
500000
600000
Visited vertices, LC heuristic
700000
800000
900000
Heuristic comparison with data grouped by agent configuration. The trend
lines being essentially the same indicates that the number of agents is not
strongly correlated with heuristic impact: regardless of method, two agents
benefit from a better heuristic as much (or little) as eight agents.
Cooperative Heuristic Search with Software Agents
Antti Halme
23/30
Results: Heuristic impact
15−puzzle runs, lengths 30−57, 1−8 agents, LC/PDB heuristic.
Visited vertices, 6−6−3 PDB heuristic
300000
A?, 1−8 agents
A!, 1−8 agents
A? lin. reg.
A! lin. reg.
250000
200000
150000
100000
50000
0
0
100000
200000
300000
400000
500000
600000
Visited vertices, LC heuristic
700000
800000
900000
Heuristic comparison with data grouped by method. The now more visible
difference in trend lines suggests that A! benefits more from the improved
1
heuristic than A?. The slope is about 81 for A?, and around 10
for A!.
Cooperative Heuristic Search with Software Agents
Antti Halme
24/30
Results: Path diversity
2
1
1
2
2
1400 1
1400 1
2
3
3
4
4
1200
3
1000
4
5
800
5
600
1000
4
800
1
1200
3
1000
5
1400
2
1200
800
5
600
6
6
6
7
7
400 7
400 7
8
8
8
8
9
9
200
600
6
400
200
9
200
9
0
1
2
3
4
5
6
7
8
1
9
2
3
4
5
6
7
8
9
1
Vanilla A∗
Solution path
2
3
4
5
6
7
8
9
1
Random A?
2
3
4
5
6
7
8
9
Cooperative A!
Visualization of A∗ , A? and A! on 8-puzzle [8 6 7 2 5 4 3 0 1]. The heat-maps are derived from a 9 × 9
self-organizing map trained on an optimal solution path of 31 steps, shown top left in the codomain.
A∗
METHOD
FREQ .
MEAN
RATIO
STD
1
648
0.11
476
2
414
0.07
363
3
269
0.04
235
4
4780
0.78
525
1
3290
0.31
1345
A?
2
3
1147 903
0.11 0.09
559
327
A!
4
5245
0.50
716
1
562
0.08
233
2
600
0.09
254
3
1006
0.15
372
4
4544
0.68
656
State visits for A∗ , A? and A!. The table gives the mean, ratio and standard deviation of state visit frequencies from
ten iterations of A∗ on four agents with Manhattan heuristic.
Cooperative Heuristic Search with Software Agents
Antti Halme
25/30
Results: Path diversity
4
4
x 10
4
x 10
x 10
2
2
2
4
4
6
6
4.5
4.5
2
4
4
4
3.5
6
3
8
10
12
12
2
2.5
10
2
1.5
14
4
6
8
10
12
14
4
6
8
10
12
14
Vanilla A∗
Solution path
16
0.5
16
0
2
16
1
14
0.5
16
0
2
1.5
1
14
16
2
12
0.5
16
2.5
10
1.5
12
1
14
3
8
2.5
1
10
3.5
6
3
8
4
4
3.5
8
4.5
2
0
2
4
6
8
10
12
14
16
0
2
Random A?
4
6
8
10
12
14
16
Cooperative A!
16 × 16 SOM visualization of A∗ , A? and A! on 54-optimal 15-puzzle [12 8 6 3 13 4 2 7 0 9 15 5 14 10 11 1].
A∗
METHOD
FREQ .
MEAN
RATIO
STD
1
661
0.00
352
2
862
0.01
1573
3
346
0.00
379
4
158124
0.99
1420
1
91459
0.43
27768
A?
2
3
32347 13256
0.15
0.06
27092 14508
A!
4
76659
0.36
4702
1
40168
0.31
29408
2
8406
0.07
8212
3
4253
0.03
1800
4
75356
0.59
1040
State visits for A∗ , A? and A! on 15-puzzle [12 8 6 3 13 4 2 7 0 9 15 5 14 10 11 1]. The table gives the
mean, ratio and standard deviation of state visit frequencies from ten iterations of A∗ on four agents with a 6-6-3
disjoint pattern database heuristic.
Cooperative Heuristic Search with Software Agents
Antti Halme
26/30
Results: Hybrid performance
I
Diminishing returns from simply adding more agents
I
I
Path diversity is essential
I
I
Secondary heuristic performance depends on it
Ideally, one would like to have the focused search
performance of A! and the diversity apparent in A?
I
I
Search overhead, new agents mostly tread on old paths
A! + A? = A00
Simple threshold combination does not appear to work
I
A! seems to skip past the very states that A? wastes time on
Cooperative Heuristic Search with Software Agents
Antti Halme
27/30
15−puzzle runs, HYBRID mode, even length groups 40−52, 4 agents, 5x, PDB heuristic.
90000
52
80000
I Hybrid A00 performance over
70000
Visited vertices for winning agent, means by group
multiple p-thresholds
60000
I Three iterations per instance per
group in the 45–54 range
50000
I A! likelihood over A? grows with
50
p to the right
40000
I
48
30000
20000
10000
46
The downward trending slopes
suggest that adding some
A? elements into A! does not
improve overall performance
44
42
40
0
1
5
9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Cutoff threshold p for A": Pr[ A! ] = p, Pr[ A? ] = 1−p.
Cooperative Heuristic Search with Software Agents
Antti Halme
28/30
Results: Discussion
I
A! outperforms both vanilla A∗ and the non-cooperative
random parallel search A∗ -variant A?
I
Nondeterministic cooperation emerging from async
message exchange was shown to be beneficial
With more agents, A! was shown to work better
I
I
I
I
Adding simple explicit randomization did not help
I
I
More robust diversity increasing mechanisms are needed
A! demonstrated extra sensitivity for heuristic improvement
I
I
Rapidly diminishing returns
Search overhead appeared to be an issue
The reason for this is unclear
SOM visualization proved thought-provoking, but lo-fi
I
More appropriate visualizations worth exploring
Cooperative Heuristic Search with Software Agents
Antti Halme
29/30
Outline
Cooperation and heuristic search
The A! algorithm
Solving n-puzzles with A!
Conclusion
Cooperative Heuristic Search with Software Agents
Antti Halme
30/30
Thesis summary
I
I
Cooperation approach to parallel computing
Heuristic search context, parallel A∗
I
I
Cooperation as a secondary heuristic
Cooperation is communication
I
I
Asynchronity, concurrency and nondeterminism
Ripple effects, implicit randomness
I
The A! algorithm
I
Empirical study, computational experiments
I
I
I
Performance, scalability
Path diversity, heuristic sensitivity
Hybrid algorithm
Cooperative Heuristic Search with Software Agents
Antti Halme
30/30
Related work
I
I
Alba, E. (Ed.). (2005). Parallel Metaheuristics. John Wiley & Sons. doi:10.1002/0471739383
I
Barbucha, D. (2012). Search Modes for the Cooperative Multi-agent System Solving the Vehicle Routing
Problem. Neurocomputing, 88, 13–23. doi:10.1016/j.neucom.2011.07.032
I
Burns, E., Lemons, S., Ruml, W., & Zhou, R. (2010). Best-First Heuristic Search for Multicore Machines.
Journal of Artificial Intelligence Research, 39, 689–743. doi:10.1613/jair.3094
I
Clearwater, S. H., Huberman, B. A., & Hogg, T. (1991). Cooperative Solution of Constraint Satisfaction
Problems. Science, 254(5035), 1181–3. doi:10.1126/science.254.5035.1181
I
Crainic, T. G., & Toulouse, M. (2008). Explicit and Emergent Cooperation Schemes for Search Algorithms.
In Proc. of the 2nd Intl. Conf. on Learning and Intelligent Optimization (LION ’07) (pp. 95–109).
Alba, E., Luque, G., & Nesmachnow, S. (2013). Parallel Metaheuristics: Recent Advances and New Trends.
Intl. Trans. in Operational Research, 20(1), 1–48. doi:10.1111/j.1475-3995.2012.00862.x
doi:10.1007/978-3-540-92695-5_8
I
Hogg, T., & Huberman, B. A. (1993). Better Than the Best: The Power of Cooperation. In 1992 Lectures in
Complex Systems (Vol. V, pp. 165–184). Addison-Wesley.
I
Hogg, T., & Williams, C. P. (1993). Solving the Really Hard Problems with Cooperative Search. In Proc. of
the 11th National Conference on Artificial Intelligence (AAAI ’93) (pp. 231–236).
I
Kishimoto, A., Fukunaga, A., & Botea, A. (2013). Evaluation of a Simple, Scalable, Parallel Best-First
Search Strategy. Artificial Intelligence, 195(0), 222–248. doi:10.1016/j.artint.2012.10.007
I
Nitschke, G. (2005). Emergence of Cooperation: State of the Art. Artificial Life, 11(3), 367–96.
I
Halme, A. (2014). Cooperative Heuristic Search with Software Agents. M.Sc. thesis, Aalto University.
doi:10.1162/1064546054407194
Cooperative Heuristic Search with Software Agents
Antti Halme
30/30