Analysis of Algorithms
Chapter - 03
Sorting Algorithms
1
This Chapter Contains the
following Topics:
1. Simple Sorts
i. Bubble Sort
ii. Insertion Sort
2. Heap sort
i.
ii.
iii.
iv.
v.
vi.
2
Heaps
The Heap Property
Maintaining the Heap Property
Building a Heap
The Heapsort Algorithm
Priority Queues
Simple Sorts
3
Bubble Sort
Sorting Problem
Input: A sequence of n numbers (a1, a2,…, an).
 Output: A permutation of n numbers (reordering)
(a'1, a'2,…, a'n) of the input sequence such that a'1≤
a'2 ≤… ≤ a'n.
Bubble Sort
It is a popular sorting algorithm
It swaps adjacent elements that are out of order
Algorithm BubbleSort (A, n)
{
for i:=1 to n-1 do
{
for j := n downto i+1 do
{
if ( A[j]<A[j-1]) then
exchange A[j] ↔ A[j-1];
}
}
}
 Illustrate the method.
 What is the worst-case complexity of Bubble sort?
4
Insertion Sort
 Insert an element to a sorted array such that the
order of the resultant array be not changed.
Algorithm InsertionSort (A, n)
{
for i:=2 to n do
{
key:=A[i];
// Insert A[i] into the sorted sequence A[1…i-1].
j:=i-1;
while ( (j>0) and (A[j]>key) ) do
{
A[j+1]:=A[j];
j:=j-1;
}
A[j+1]:=key;
}
}
 What is the worst-case complexity of Insertion
sort?
5
Analyzing Simple Sorts
 Bubble sort
-O(n2)
- Very simple code
Insertion sort
- Slightly better than bubble sort
- Fewer comparisons required.
- Also O(n2)
 Bubble Sort and Insertion Sort
- Use when n is small, (n ~10);
- Simple code compensates for low efficiency.
n^2 and n log n
2500
2000
Time
1500
n log n
n^2
1000
500
0
0
10
20
30
n
6
40
50
60
Heap sort
7
Heaps
 A heap can be seen as a complete binary tree:
16
14
10
8
2
7
4
9
1
 What makes a binary tree complete?
 Is the example above complete?
8
3
Heaps (Contd.)
 A heap can be seen as a complete binary tree:
16
14
10
8
2
7
4
1
9
1
1
3
1
1
 They are called “nearly complete” or “full” binary
trees.
 We can think the unfilled slots as null pointers.
9
1
Heaps (Contd.)
 In practice, heaps are usually implemented as
arrays.
 To represent a complete binary tree as an array:
 The root node is A[1]
 Node i is A[i]
 The parent of node i is A[i/2] (note: integer
divide)
 The left child of node i is A[2i]
 The right child of node i is A[2i + 1]
 For example, the array
A = 16 14 10 8 7 9 3 2 4 1
can be represented as the following heap.
16
14
10
8
2
10
7
4
1
9
3
Referencing Heap Elements
 So…
Parent(i)
{
return i/2;
}
Left(i)
{
return 2*i;
}
right(i)
{
return 2*i + 1;
}
11
The Heap Property
 Heaps also satisfy the max-heap property:
 A[Parent(i)]  A[i] for all nodes i > 1
 In other words, the value of a node is at most
the value of its parent.
 In min-heap,
 A[Parent(i)] ≤ A[i] for all nodes i > 1
 In other words, the value of a parent node is at
most the value of its child nodes.
 Where are the largest elements in a max-heap
and min-heap stored?
 Definitions:
 The height of a node in a head = the number of
edges on the longest simple downward path
from the node to a leaf.
 The height of the heap = the height of its root.
 Since a heap of n-elements is based on a
complete binary tree, its height is Θ(lg n)
 The basic operations on heaps run in time at
most proportional to the height of the tree and
thus take O(lg n).
12
Maintaining the Heap Property
 MaxHeapify(): maintains the max-heap
property.
 Given: a node i in the heap with children l
(left) and r (right).
 Given: two subtrees rooted at l and r,
assumed to be heaps.
 Problem: The subtree rooted at i may
violate the heap property (How?)
 Action: let the value of the parent node
A[i] “float down” so subtree rooted at
index i becomes a max-heap.
 What do you suppose will be the basic
operation between i, l, and r?
13
The Algorithm MaxHeapify()
Algorithm MaxHeapify(A, i)
{
l := Left(i);
r := Right(i);
if ((l <= heap_size(A)) and (A[l] > A[i])) then
largest := l;
else
largest := i;
if ((r <= heap_size(A)) and (A[r] > A[largest])) then
largest := r;
if (largest ≠ i) then
{
exchange A[i]↔A[largest];
MaxHeapify(A, largest);
}
}
14
Illustration by an Example
16
4
10
14
2
7
8
1
9
3
A = 16 4 10 14 7 9 3 2 8 1
16
4
10
14
2
15
7
8
1
9
3
A = 16 4 10 14 7 9 3 2 8 1
Example (Contd.)
16
4
10
14
2
7
8
1
9
3
A = 16 4 10 14 7 9 3 2 8 1
16
14
10
4
2
16
7
8
1
9
3
A = 16 14 10 4 7 9 3 2 8 1
Example (Contd.)
16
14
4
2
10
7
8
9
3
1 A = 16 14 10 4 7 9 3 2 8 1
16
14
4
2
17
10
7
8
9
3
1 A = 16 14 10 4 7 9 3 2 8 1
Example (Contd.)
16
14
8
2
10
7
9
3
1 A = 16 14 10 8 7 9 3 2 4 1
4
16
14
8
2
18
10
7
4
9
3
1 A = 16 14 10 8 7 9 3 2 4 1
Example (Contd.)
16
14
10
8
2
7
4
9
1
A = 16 14 10 8 7 9 3 2 4 1
19
3
Analyzing MaxHeapify()
 The running time of the algorithm on a
subtree of size n rooted at given node i is
the
(1) time to fix up the relationships
among the elements A[i], A[l] and A[r],
Plus the time to run MaxHeapify() on
a subtree rooted at one of the children
of node i.
 If the heap at i has n elements, how many
elements can the subtrees at l or r have?
Answer: 2n/3
 The worst case occurs when the last row
of the tree is exactly half full.
 So time taken by MaxHeapify() is given
by
T(n)  T(2n/3) + (1)
 By case 2 of the Master Theorem,
T(n) = O(lg n)
 Alternatively, we can characterize the
running time of MaxHeapify() on a node
of height h as O(h).
20
Building a Heap
 We can build a heap in a bottom-up manner
by running MaxHeapify() on successive
subarrays:
 Fact: for array of length n, all elements in
range A[n/2 + 1 .. n] are heaps (Why?)
 Walk backwards through the array from
n/2 to 1, calling MaxHeapify() on each
node.
 Order of processing guarantees that the
children of node i are heaps when i is
processed.
Algorithm BuildMaxHeap(A)
{
heap_size(A) := length(A);
for i := length[A]/2 downto 1 do
MaxHeapify(A, i);
}
 Work through the example
A = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7}
21
Analyzing BuildMaxHeap
 Each call to MaxHeapify() takes O(lg n)
time.
 There are O(n) such calls (specifically,
n/2)
 Thus the running time is O(n lg n).
 This is an upper bound, though correct,
but is not asymptotically tight.
 Our tighter analysis relies on the
properties that an n-element heap has
height lg n, and at most |n/2h+1| of any
height h.
 The time required by MaxHeapify() when
called on a node of height h is O(h).
 So we can express the total cost of
BuildMaxHeap() as being bounded from
above by the following calculation.
22
Analyzing BuildMaxHeap (Contd.)
 lg n  h 
 n 

 2 h 1 O (h)  O n  2 h .

h 0 
 h 0 
lg n 

x
, for x  1,
2
(1  x)
k 0
Substituting x  1 / 2, we get the last summation
Since we have that  kxk 

h
1/ 2

2

h
2
(1  1 / 2)
h 0 2
Thus, the running time of BuildMaxHe ap ()
as
can be bounded as
 lg n  h 
  h 
O n  h   O n h   O (n).
 h 0 2 
 h 0 2 
Hence, we can build a max  heap from an
unordered array in linear time.
23
Heapsort
 The Heapsort algorithm starts by using
BuildMaxHeap() to build a max-heap on the
input array A[1..n], where n=length[A].
 The maximum element of the array is stored
at the root A[1].
 Swap A[1] with element at A[n] to put the
maximum element into its correct position.
 Then discard the node n from the heap,
which will decrement the heap_size[A] by
one.
 The children of the root remain max-heap,
but the new root element may violate the
max-heap property.
 So, restore the max-heap property at A[1] by
calling MaxHeapify(A,1).
 This will leaves a max-heap in A[1..(n-1)].
 Always swap A[1] for A[heap_size(A)], and
repeat the process for the heap_size(A)
down to 2.
24
The Heapsort Algorithm
Algorithm Heapsort(A)
{
BuildMaxHeap(A);
for i := length(A) downto 2 do
{
exchange A[1] ↔ A[i];
heap_size(A) := heap_size(A) - 1;
MaxHeapify(A, 1);
}
}
 Work through the example
A = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7}
 Analyzing the heapsort algorithm
 The call to BuildMaxHeap() takes O(n) time.
 Each of the n - 1 calls to MaxHeapify() takes
O( lg n) time.
 Thus the total time taken by Heapsort()
= O(n) + (n - 1) O(lg n)
= O(n) + O(n lg n)
= O(n lg n)
25
Priority Queues
 One of the most popular application of a heap
is an efficient priority queue.
 A priority queue is a data structure for
maintaining a set S of elements, each with an
associated value called a key.
 There are two kinds of priority queues:
 Max-priority queues and
 Min-priority queues.
 We will focus how to implement max-priority
queues, which are based on max-heaps.
 A max-priority queue supports the following
operations:
26
 Insert(S, x): Inserts the element x into the set S.
This operation could be written as S:= S U {x}.
 Maximum(S): Returns the element of S with the
largest key.
 ExtractMax(S): Removes and returns the
element of S with the largest key.
 IncreaseKey(S, x, k): Increases the value of
element x’s key to the new value k, which is
assumed to be at least as large as x’s current
key value.
Priority Queues (Contd.)
 Application of max-priority queues:
 Scheduling jobs on a shared computer.
 It keeps the track of jobs to be performed and their
relative priorities.
 When a job is finished or interrupted, the highest priority
job is selected from those pending using ExtractMax().
 A new job can be added to the queue at any time using
Insert().
 A min-priority queue supports the operations:
 Insert(), Minimum(), ExtractMin(), and DecreaseKey().
 Application of min-priority queues:
 It can be used as event-driven simulator.
 The items in the queue are events to be simulated, each
with an associated time of occurrence that serves as its
key.
 The events must be simulated in order of their time of
occurrences, because the simulation of an event can
cause other events to be simulated in the future.
 The simulation program uses ExtractMin() at each step
to choose the next event to simulate.
 As new events are produced, they are inserted into the
min-priority queue using Insert().
27
Operations
 The procedure HeapMaximum() implements the
Maximum() operation in Θ(1) time.
Algorithm HeapMaximum(A)
return A[1];
 The procedure HeapExtractMax() implements the
ExtractMax() operation as follows:
Algorithm HeapExtractMax(A)
{
if (heap_size(A) < 1) then
error “heap underflow”;
max := A[1];
A[1] := A[heap_size(A)];
heap_size(A) := heap_size(A) – 1;
MaxHeapify(A, 1);
return max;
}
 The running time of HeapExtractMax() is O(lg n),
since it performs only a constant amount of work
on top of the O(lg n) time for MaxHeapify().
28
Operations (Contd.)
 The
procedure
HeapIncreaseKey()
implements the IncreaseKey() operation.
 The priority-queue element whose key is
to be increased is identified by an index i
into the array.
 The procedure first updates the key of
element A[i] to its new value.
 Increasing the key of A[i] may violate the
max-heap property.
 So, the procedure then traverses a path
from this node towards the root to find a
proper place for the newly increased key.
 During this traversal, it repeatedly
compares an element to its parent,
exchanging their keys and continuing if the
element’s key is larger, and terminating if
the element’s key is smaller, since the
max-heap property now holds
29
Operations (Contd.)
Algorithm HeapIncreaseKey(A, i, key)
{
if (key < A[i]) then
error “new key is smaller than current key”;
A[i] := key;
while ((i > 1) and (A[Parent(i)] < A[i]) do
{
exchange A[i] ↔ A[Parent(i)];
i := Parent(A);
}
}
 Since the path traced from the node updation
statement to the root has length O(lg n).
 The running time of HeapIncreaseKey() on an nelement heap is O(lg n).
 Illustrate the procedure.
30
Illustration
16
14
10
8
7
9
3
i
2
4
1
(a) The max-heap with a node whose index i is red labeled.
16
14
10
8
7
9
i
2
15
1
(b) This node has its key updated to 15.
31
3
Illustration (Contd.)
16
14
i
10
15
2
7
8
9
3
1
(c) After one iteration of the while loop, the node and its parent
have exchanged keys, and the index i moves up to the parent.
16
i
15
10
14
2
7
8
9
3
1
(d) The max-heap after one more iteration of the while loop. At
this points, A[Parent(i)] ≥ A[i]. The max-heap property now holds
and the procedure terminates.
32
MaxHeapInsert()
 The procedure MaxHeapInsert() implements
the Insert() operation.
 It takes as an input the key of new element to
be inserted into max-heap.
 The procedure first expands the max-heap by
adding to the tree a new leaf whose key is -∞.
 Then it calls HeapIncreaseKey() to set the key
of this new node to its correct value and
maintain the max-heap property.
Algorithm MaxHeapInsert(A, key)
{
heap_size(A) := heap_size(A) + 1;
A[heap_size(A)] := -∞;
HeapIncreaseKey(A, heap_size(A), key);
}
 The running time of MaxHeapInsert() on an nelement heap is O(lg n).
 In summary, a heap can support any priorityqueue operation on a set of size n in O(lg n)
time.
33
End
of
Chapter - 03
34