(Ab-) using Round Robin
Databases with Perl
Mike Schilli, Yahoo!
06/18/2008
1
Overview
What are Round Robin Databases?
 RRD and Perl – which module to use?
 Practical Examples

2
Round Robin Databases
Archives of fixed size for unlimited data
 Overwrite old spots if full

3
Round Robin Archives
start
4
Round Robin Archives

Overwrite if full
5
Round Robin Archives

RRDs with multiple archives with
different granularity
6
Data Flow with rrdtool
Data Source
Archive
7
Data Source Conversions
8
Steps and Heartbeat
9
Steps and Heartbeat
10
Steps and Heartbeat
11
Generous Heartbeat
12
Generous Heartbeat
13
Steps and Heartbeat

Long story short:
Heartbeat > Step
if you’re roughly feeding in “step” intervals.

Example:
Step=300 (5 min)
 Heartbeat=600

14
Feeding PDPs to the Archive
15
Feeding PDPs to the Archive
Consolidation Functions (MAX,
AVERAGE, etc.)
 xfiles factor (allowed percentage of
missing values)

16
17
Simple Example
Web Orders:
0:00
1:00
2:00
3:00
4:00
5:00
100
110
120
130
140
150
18
Simple Example
19
Simple Example
rrdtool 'create' 'myrrdfile.rrd' '--start' '1211871600' '--step'
'3600' 'DS:orders:GAUGE:7200:U:U'
'RRA:MAX:0.5:1:100'
rrdtool 'update' 'myrrdfile.rrd' '1211875200:100'
rrdtool 'update' 'myrrdfile.rrd' '1211878800:110'
rrdtool 'update' 'myrrdfile.rrd' '1211882400:120'
rrdtool 'update' 'myrrdfile.rrd' '1211886000:130'
rrdtool 'update' 'myrrdfile.rrd' '1211889600:140'
rrdtool 'update' 'myrrdfile.rrd' '1211893200:150'
rrdtool 'graph' 'mygraph.png' '--vertical-label' 'Orders per
Hour' '--end' '1211893200' '--start' '1211871600'
'DEF:draw1=myrrdfile.rrd:orders:MAX'
'AREA:draw1#0000FF:Web Orders
20
RRDTool
Arcane syntax
 Perl API RRDs.pm

Pretty close to RRDTool syntax
 Calls rrdtool under the hood

21
To the Rescue: RRDTool::OO
CPAN Module RRDTool::OO
 Takes the drudgery out of the RRD
syntax
 Does what you want by default
 Translates between Perl and RRD
(watch it with debug on)
 Named parameters instead of fixed order

22
Simple Example
#!/usr/local/bin/perl -w
use strict;
use RRDTool::OO;
use DateTime;
my $rrd = RRDTool::OO->new(
file => "myrrdfile.rrd" );
23
Simple Example
my $start_dt = DateTime->today(
time_zone => "local" );
my $dt = $start_dt->clone();
24
Simple Example
$rrd->create(
step
=> 3600,
start
=> $start_dt->epoch(),
data_source => { name
=> "orders",
type
=> "GAUGE",
},
archive
=> { rows => 100 }
);
25
Simple Example
for my $hour (0..5) {
$dt->add(hours => 1);
$rrd->update(
time => $dt->epoch(),
value => 100 + $hour*10,
);
}
26
Simple Example
$rrd->graph(
image
=> "mygraph.png",
vertical_label => 'Orders per Hour',
start
=> $start_dt->epoch(),
end
=> $dt->epoch(),
draw
=> {
type
=> "area",
color => '0000FF',
legend => "Web Orders",
}
);
27
Simple Example
28
RRDTool Commands
rrdtool 'create' 'myrrdfile.rrd' '--start' '1211871600' '--step'
'3600' 'DS:orders:GAUGE:7200:U:U'
'RRA:MAX:0.5:1:100'
rrdtool 'update' 'myrrdfile.rrd' '1211875200:100'
rrdtool 'update' 'myrrdfile.rrd' '1211878800:110'
rrdtool 'update' 'myrrdfile.rrd' '1211882400:120'
rrdtool 'update' 'myrrdfile.rrd' '1211886000:130'
rrdtool 'update' 'myrrdfile.rrd' '1211889600:140'
rrdtool 'update' 'myrrdfile.rrd' '1211893200:150'
rrdtool 'graph' 'mygraph.png' '--vertical-label' 'Orders per
Hour' '--end' '1211893200' '--start' '1211871600'
'DEF:draw1=myrrdfile.rrd:orders:MAX'
'AREA:draw1#0000FF:Web Orders
29
Where RRDtool shines
Graph time-based data
 Don’t worry about scaling or axis text
 Stacked graphs

30
Where RRDtool sucks
Can’t add data for past events
 Can’t add data twice at the same
timestamp

31
Pitfalls
Make sure that the first update happens
after RRD “start” time
 Updates need to happen at increasing
timestamps
 On large step sizes (1 year), limit the
number of rows to fit within < 2030.

32
Debugging with Log4perl
use Log::Log4perl qw(:easy)
Log::Log4perl->
easy_init( $DEBUG );
33
Log4perl Output
rrdtool 'create' 'myrrdfile.rrd' '--start' '1211871600' '--step'
'3600' 'DS:orders:GAUGE:7200:U:U'
'RRA:MAX:0.5:1:100'
rrdtool 'update' 'myrrdfile.rrd' '1211875200:100'
rrdtool 'update' 'myrrdfile.rrd' '1211878800:110'
rrdtool 'update' 'myrrdfile.rrd' '1211882400:120'
rrdtool 'update' 'myrrdfile.rrd' '1211886000:130'
rrdtool 'update' 'myrrdfile.rrd' '1211889600:140'
rrdtool 'update' 'myrrdfile.rrd' '1211893200:150'
rrdtool 'graph' 'mygraph.png' '--vertical-label' 'Orders per
Hour' '--end' '1211893200' '--start' '1211871600'
'DEF:draw1=myrrdfile.rrd:orders:MAX'
'AREA:draw1#0000FF:Web Orders
34
Example: Stock Portfolio
35
Example: Measure Power Usage
Mastech MAS-345
 Serial Interface (RS-232)

36
Example: Measure Power Usage
37
Example: Measure Power Usage
38
Aberrant Behaviour
39
The End
Thanks!
40
References





RRDTool::OO on CPAN
RRD::Simple on CPAN
Portfolio Rendering article:
http://perlmeister.com/talks/lm-200805.pdf
RRDTool article: http://www.linuxmagazine.com/issue/44/Perl_RDDtool.pdf
RRDTool article (in German): http://www.linuxmagazin.de/Artikel/ausgabe/2004/06/perl/perl.html
41
Bonus Slides
42
Aberrant Behaviour
Holt Winter Algorithm
 Data Smoothing/Prediction

43
Aberrant Behaviour
Holt Winter Algorithm
 RRAs:

HWPREDICT
 SEASONAL
 DEVSEASONAL
 DEVPREDICT
 FAILURES


RRA:HWPREDICT:rows:alpha:beta:sea
sonal_period
44
Aberrant Behaviour

alpha (baseline): 0..1
1: recent observations carry greater weight
 0: past observations carry greater weight

beta (slope): 0..1
 gamma (seasonal): 0..1
 failures defaults to 7

45
Aberrant Behaviour

Example:
alpha = 0.50
 beta = 0.50
 gamma = 0.01
 seasonal_period = 2
 threshold = 7
 window_length = 9

46