Where I Want to be.....
+
2
What I'll Settle For.....
3
Requirements

Rock solid stability

Automated node addition (discovery)

Scales horizontally

Service dependency models

Easy to write plugins

Promotes sane workflows

Unified front end view

Flexible configuration
4
Tool Stack
5
What????
but #monitoringsucks and #ihatenagios
How could you?
6
In defense of Nagios

Been around since 1996

Has Service dependencies

Easy to write plugins

Easy-ish to troubleshoot

ROCK SOLID
7
Valid attacks on Nagios

No automated discovery

It's complicated to setup

Text files – really?

Front end won't win any beauty contests

Development is slow

Stats collection is a PITA
8
Solutions

Use Icinga!

Use Puppet to auto configure

Stats – leave it to graphite. It's really good at that

Big boys and girls learn their tools
9
Icinga

Fork of Nagios

Configurations are compatible

More solid architecture ( core, API, Web, IDODB )

Nice front end, nice mobile front end

Can use NRPE
10
High Level View
11
Configure Icinga Servers
using Puppet Standard Types
12
Things to configure with Standard Types

icinga.cfg (file)
=> icinga main config file

Apache icinga.conf (file)
=> http access to each server

cgiauth.cfg (file)
=> cgi access

cgi.cfg (file)
=> options, users

templates.cfg (file) got lazy => use for basic classes

idomod.cfg (template)
=> template for hostname to DB
13
Configure Icinga using Nagios Types
14
Puppet Nagios Types

nagios_command

nagios_hostgroup

nagios_contact

nagios_service

nagios_contactgroup

nagios_servicedependency

nagios_host

nagios_serviceescalation

nagios_hostdependency

nagios_serviceextinfo

nagios_hostescalation

nagios_servicegroup

nagios_hostextinfo

nagios_timeperiod
15
Configuring Hosts
16
Overview
17
Detailed Overview
18
Store Configs


Store puppet info in a DB
Retrieve information from
DB

Share info across nodes

Use thin_storeconfigs

Set up on puppet master
19
Exporting Nagios_host Resources

Export = Save to DB

Use facter for dynamic data

PRO TIP: use ENC

PRO TIP: use targets

PRO TIP: hostgroups

PRO TIP: use tags
20
PRO TIP: Use your ENC
21
PRO TIP: use targets




Use cfg_dir in icinga.cfg
Create a unique file per host
or service
Addition and removal are
now super easy
Also default dirs are in a
horrible place /etc/nagios
22
PRO TIP: hostgroups



Add machines to a
hostgroup
Add services to a hostgroup
New machines inherit all of
the services associated with
a hostgroup
23
PRO TIP: use tags
Tags allow you to filter
resources so that you only
realize those resources that
you need
24
Configuring Services/Commands
25
Icinga Services

OR 'Stuff I want to monitor'

Associate with a hostgroup

Use a target
26
Icinga Commands

OR 'What actually gets run'

Use Macros to set paths in resource.cfg
27
Dependencies
28
PRO TIP: Dependencies

Unreliable services

Cut down on the number of alerts

Tell me what's really wrong

Route alerts accordingly
29
Nagios_servicedependencies
30
NRPE
31
NRPE

Runs on client

Secured via SSL

Has ACLs

Runs as nobody

Can run commands

Useful for other things...
32
Configuring NRPE
33
NRPE Checks
34
Plugins
35
exchange.nagios.org
36
Writing Plugins

Write in any language

Output 1 line to stdout


NRPE/Icinga/Nagios all use exit
codes to determine status
Run by hand to check
37
Workflows
38
Watching Monitoring
39
Scheduling Downtime
40
Filtering
41
Alerting
42
#monitoringisawesome

REMOVE unreliable checks

Just MONITOR – don't bolt on - especially stats

TIER your monitoring

Use timeperiods for sanity

Delegate responses

Use dependencies to pin down problems quickly

Work smart
43
44
Resources

Icinga
http://icinga.org

Puppet
http://docs.puppetlabs.com/references/latest/type.html#nagioscommand

NRPE
http://nagios.sourceforge.net/docs/3_0/addons.html

IRC
##infra-talk, #icinga, #puppet

Contact
[email protected], @maguec, #gaijin (freenode), http://blog.mague.com

Thanks
Yvonne Kong, Michael Catlin, Juan Ortega, Anthony Kong, Puppet Labs, Icinga Team
45