Example
Cost based Query Optimization in
(Distributed) Databases
• Cost based QO: parameters
• Cost functions
• Distributed joins:
searching the state space
- brute force
- hill climing
• Joins using semi-joins
Part of Geo-DB
Land ( name, l_id, …)
Berg (name, hoehe, ..)
geo_Berg ( l_id, berg, …)
umfasst (land, kontinent, …) /* land = l_id
sunLand (..)
sunUmfasst (..)
sungeo_Berg (..)
S_db
t_db
umfasst(..)
berg (..)
geo_Berg
(..)
A complex query: example
“for each country in America find highest mountain and height”
slides partially based on H. Garcia-Molina; P. Valduriez
hs / FUB dbsII-03-14DDBQueryOpt-2
Example Query
The query plan
SELECT DISTINCT l.name, b.name, b.hoehe, 'm‘
FROM sunland l, geo.berg b, sunumfasst u,
geo.geo_berg gb
WHERE u.kontinent = 'Amerika'
AND u.land = l.l_id
AND u.land = gb.l_id
AND gb.berg = b.name
AND b.hoehe = (SELECT max(b1.hoehe)
FROM geo.berg b1, sungeo_berg gb1
WHERE b1.name = gb1.berg
AND gb1.l_id = u.land)
ORDER BY 3 DESC
hs / FUB dbsII-03-14DDBQueryOpt-3
hs / FUB dbsII-03-14DDBQueryOpt-4
Cost
Cost based query optimization
Optimization problems require
Search space
Main influence: size of (intermediate) results
… as in centralized system
Attribute length length(a)
Search strategy
number of distinct attribute (a) values of relation R |R.a|
max(a), min( a): number of distinct values of attribute a
Optimization function
cardinaltiy of a |dom a|
cardinality of the relation (or fragment) |R|
Query optimization
Search space: Query execution plans (usually
many for a query )
Search strategy: typical is dynamic programming
newer technique: randomized
Cost function or response time
hs / FUB dbsII-03-14DDBQueryOpt-5
Important: Selectivity
Join : sfjoin = | R
S | / |R| * |S|
⇒ σa=val (R) = |R| / |R.a|
value set S.A of a: sf (S.A) = sf (a∈ S.A) = |S.A| * / |S.a|
value: sf ( a=val) = 1 / |R.a|
hs / FUB dbsII-03-14DDBQueryOpt-6
1
Cost parameters
Cost parameters
Projection
|R.a| = |R |
Join
if a is key
Special case: A is a key of R and B is a foreign key of S;
|R
Cartesian Product
|R × S | = |R | * |S |
A=B
Union
|R
upper bound:
|R ∪ S | = |R | + |S |
lower bound:
|R ∪ S | = max { |R|, |S | }
|R–S |= |R|
lower bound:
0
S | = sf join * |R | * |S |
(difficult to determine)
Semijoin
|R
Set Difference
upper bound:
S | = |S |
general:
A
S | = sf semi (S.A) * | R |
where
sf semi (R
A
S) = sf semi (S.A) = |S.A | / |dom (A) |
approximation, exact for A key in S, foreign key in R, sfsemi = 1
hs / FUB dbsII-03-14DDBQueryOpt-7
Cost parameters …
hs / FUB dbsII-03-14DDBQueryOpt-8
Histograms for attribute values
… in practice
Histograms
DBS use object level statistics
Assumption "uniform distribution of data" mostly wrong
Dynamically collected histograms allow for much better
predictions
Different styles:
- e.g. |R| and number of blocks of R,
- # of B-tree levels if indexed
- column statistics for each column:
min, max value, |R.a| , column histograms
System statistics
- CPU speed
- IO – performance, IO / CPU ratio
Statistics management
Offline gathering
- sampling: use only a small number of rows
- monitoring: refresh if # updates > threshhold
- histogram selection for atttribute a
dependent on skew an frequeny of use of an attribute
Dynamic sampling during optimization
1 10 20 30 40 50 60
width-balanced
1
hs / FUB dbsII-03-14DDBQueryOpt-10
Plan A
# IOs may not be best metric
Transmission time may dominate
work
at site
60
Cost measure and parallelism
Cost functions
Estimation of set (intermediate, result) size as in
central, but…
work
at site
39 41 45
height balanced
Advantage of height balanced historgram for attribute:
more exact approximation in ranges with many values
Disadvantage?
hs / FUB dbsII-03-14DDBQueryOpt-9
Cost functions in distributed DBS
18 23
100 IOs
answer
Plan B
site 1
50 IOs
site 2
70 IOs
site 3
50 IOs
Influence on response time: max (I/O), not ∑ I/O
Same holds for CPU time
Another reason why plain IOs not enough:
Parallelism
hs / FUB dbsII-03-14DDBQueryOpt-11
hs / FUB dbsII-03-14DDBQueryOpt-12
2
Cost metrics in distributed environment
Cost / Time in distributed (DB) systems
Cost metrics
Parallel and distributed environment
additive:
total cost = total I/O + total CPU + total transmission
Data distribution costs / time
Response time
Contention
much complexer:
Q
Q
Q
Start up costs (for parallel operation)
Task 1
task scheduling
dependencies
do not add!
memory, disk, network,…
Assembling result
Task 2
Task 3
hs / FUB dbsII-03-14DDBQueryOpt-13
Example: Response time
hs / FUB dbsII-03-14DDBQueryOpt-14
Searching the search space
Search space: all valid query execution plans
Problem:
How to transform plans systematically?
Site 1
Site 2
Site 3
Site 4
Startup
DistriSearching
bution + send results
proc.
Final
hs / FUB dbsII-03-14DDBQueryOpt-15
Strategies
hs / FUB dbsII-03-14DDBQueryOpt-16
Simple Join Ordering
Exhaustive (with pruning)
Hill climbing (greedy)
Query separation
Two relations
|R| < |S|
R
S
|S| < |R|
Exhaustive
Multiple relations difficult, too many alternatives.
systematic generation of all query plans
Q expensive!
inacceptable for ad hoc queries,
ok for “canned” queries ?
Q O(n!) different joins of n relations
hs / FUB dbsII-03-14DDBQueryOpt-17
Compute the cost of all alternatives and select the best one.
Necessary to compute the size of intermediate relations
which is difficult.
Use heuristics
hs / FUB dbsII-03-14DDBQueryOpt-18
3
Join Ordering
Join ordering
Most important in DDBS : Joins
join n relations at n sites R1
Exhaustive Search
R2
…..
Rn
Linear join tree
if only linear join trees:
O(2n)
“Bushy” join tree
Goal: join R1
effective
R2
….
Rn most cost-
First step: find best access strategy for R1,…Rn
depending on join attribute,
e.g. full scan, index access on join attribute, …
Second step:
For each permutation of (Ri1,…,Rin) calculate
Ri1
….
Rin
Prune tree where possible
hs / FUB dbsII-03-14DDBQueryOpt-19
Exhaustive search: join ordering
A
R
Example:: join
R
R
S
Hill climbing: join order
B
S
T
|R|>|S|>|T|
S
R×T
2
S
R
S
1
T
T
T
S
T×R
2
(S
R)
1
T
ship S
semi
to R
join to S
(T
ship T
S)
R
semi
join
1 Prune because cross-product not necessary
2 Prune because larger relation first
e.g. S
T has higher cost than T
Expl. by H. Garcia-Molina
S
hs / FUB dbsII-03-14DDBQueryOpt-21
Hill climbing: join order
Site
Size
R
1
10
S
2
20
T
3
30
V
4
40
Hill climbing:
Start with initial solution
Change solution locally with maximal benefit
(e.g. change order of join execution)
Stop if for every possible change the benefit is
negative
Task:
R
S
T
V
A
B
R
Which goal?
S
C
T
V
Minimal data transmission?
Minimal CPU (e.g. # comparions)
Minmal IO ?
Expl. by H. Garcia-Molina
hs / FUB dbsII-03-14DDBQueryOpt-22
Hill climbing: join order
Goal: minimize data transmission!
Rel
hs / FUB dbsII-03-14DDBQueryOpt-20
tuple size = 1
Local transformation
Consider sending each relation to neighbor:
e.g.:
Initial plan: Send all relations to one site. Which one?
R
1
4
4
S
2
1
R S
R
2
To site 1: cost=20+30+40=90
To site 2: cost=10+30+40=80
To site 3: cost=10+20+40=70
To site 4: cost=10+20+30=60
hs / FUB dbsII-03-14DDBQueryOpt-23
hs / FUB dbsII-03-14DDBQueryOpt-24
4
Hill climbing: join order
Assume: Size
Hill climbing: join order
R
S = 20
S
T=5
T
V=1
Option (b)
Option (a)
R
10
4
S
1
4
20
2
20
R
R
10
1
Cost in initial plan= 30
4
S R
2
20
1 S
cost = 30
2
R
10 20
1
S
4 S
20
2
cost = 40
Worse off!
cost = 30
No win!
hs / FUB dbsII-03-14DDBQueryOpt-25
Hill climbing: join order
hs / FUB dbsII-03-14DDBQueryOpt-26
Hill climbing: join order
Option (d): Join S with right neighbor in join graph
Option (c)
4
S
2
10 30
4
T S
T
3
S
5
30
2
•Hill climbing: join order
T
cost = 50
2
3
4
4 T
20 30
5
20
3
2 S
cost = 50
cost = 35
Win!
Bigger Win!
Plan 1:
P1a: S (2 → 3)
α=S T
compute answer at site 4
hs / FUB dbsII-03-14DDBQueryOpt-27
hs / FUB dbsII-03-14DDBQueryOpt-28
Hill climbing may miss best plan!
Example R
- Treat α = S
R
1
4
PB:
T as relation
R
α
α
vs
1
3
1
3
α
R
T
V
β (4 → 2)
β’= β
4
S
T (3 → 4)
β=T
4
S
β’ (2 → 1)
β”= β’
α
R
3
hs / FUB dbsII-03-14DDBQueryOpt-29
3
cost = 25
P1b: R (1 → 4)
α (3 → 4)
Hill climbing: join order – repeated local search
S T
R
V:
best plan could be
V
➽30
➽1
1
R
➽1
β
β’
2
T
3
S
T
Costs could be low
because is
very selective
β” (1 → 4)
Compute answer
4
β’’
➽1
33 = total
hs / FUB dbsII-03-14DDBQueryOpt-30
5
Join in DDB: using filters
Join by Semi joins
Join with filters
Use following identity:
- Do not transfer relations
R
- Transfer those attribute values of R needed to
perform join with S
- Use semi-join to find "join-partners"
R
S:
A
S
S
=
(R
=
R
=
(R
A
A
A
S)
(S
S)
criterion: |R| > | ∏ A S| + | R
S.A
R:
A
project on a
hs / FUB dbsII-03-14DDBQueryOpt-31
S
A
R)
A
(S
A
A
R)
S|
A
S.A
Philosophy: distributed join is hardest
join with join keys only,
pick up other attributes needed later
hs / FUB dbsII-03-14DDBQueryOpt-32
Conclusion
Distributed Query Optimization is hard
Cost based optimizers state of the art
Huge number of parameters
Æ concentrate on important ones,
e.g. data transmission
Heuristics inevitable
Which goal ?
Some systems allow to adjust optimizer
for minimal response or minimal cost
Some systems allow "hints"
Æ not only DB implementer must know about QO
but also administrator / application programmer
hs / FUB dbsII-03-14DDBQueryOpt-33
6