Monitor your infrastructure
with the Elastic Beats
Monica Sarbu
Monica Sarbu
Team lead, Beats team
Email: [email protected]
Twitter: @monicasarbu
2
Monitor your servers
Apache logs
@monicasarbu
3
Monitor your servers
Apache logs
memory %
CPU %
@monicasarbu
4
Monitor your servers
Apache logs
Apache metrics
@monicasarbu
memory %
CPU %
5
Monitor your servers
Apache logs
Apache metrics
memory %
CPU %
HTTP transactions
@monicasarbu
6
Multiple data types, one storage
Apache logs
Apache metrics
memory %
CPU %
HTTP transactions
@monicasarbu
7
Scalable from day 1
@monicasarbu
8
Beats are lightweight shippers
that collect and ship all kinds of
operational data to Elasticsearch
Kibana
Elastic
Stack
Elasticsearch
Beats
@monicasarbu
Logstash
10
The Beats
30+ other community Beats shipping
@monicasarbu
11
Filebeat
12
tail -f
tail -f
over the Network
tail -f
over the Network
with extra powers
http://www.clipartpanda.com/clipart_images/witches-clip-art-6144127
Multiline
JSON logs
Filtering
Send raw log lines
{
}
message: “55.3.244.1 GET /index.html 15824 0.043”,
…
@monicasarbu
17
Parse log lines by defining grok patterns
I
N
G
E
S
T
@monicasarbu
or
18
Grok patterns
{
}
message: “55.3.244.1 GET /index.html 15824 0.043”,
…
%{IP:client} %{WORD:method} %{URIPATHPARAM:request}
%{NUMBER:bytes} %{NUMBER:duration}
@monicasarbu
19
After parsing
{
}
message: “55.3.244.1 GET /index.html 15824 0.043”
client: “55.3.244.1”,
method: “GET”,
request: “/index.html”
bytes: 15824,
duration: 0.043
…
@monicasarbu
20
Handle back-pressure
Why back-pressure is key?
@monicasarbu
22
Synchronous sending
registry file
acked
read
read
stream of log lines
batch of messages
ack
@monicasarbu
23
Filebeat adapts its speed
automatically to as much as the
next stage can process
When next stage is down …
• Filebeat patiently waits
• Log lines are not lost
• It doesn’t allocate memory
• It doesn’t buffer log lines on disk
@monicasarbu
25
At-Least-Once
delivery
No such think as “exactly once”
#velo
@monicasarbu
27
batch of messages
ack
batch of messages
#velo
same b
atch of
messag
es
ack
duplicates!
@monicasarbu
28
Filebeat
Collect container logs
29
Docker logging drivers
https://docs.docker.com/engine/admin/logging/overview/
@monicasarbu
30
001 Gelf driver + Logstash
Pros:
Cons:
• logs send directly to
Logstash
• UDP based, no delivery
guarantees, no
congestion control
@monicasarbu
31
010 json-file driver + Filebeat
Pros:
Cons:
• Simple to setup as it’s the
default driver
• json-file driver can slow
down Docker container
• Easy to add container
metadata (name, labels,
etc.)
• `docker logs` works
@monicasarbu
32
011 Syslog driver + Syslog server + Filebeat
Pros:
Cons:
• Good control over the
path where the files are
written, rotation strategies,
etc.
• you need to manage the
syslog server
• metadata is serialized as
string, needs to be deserialized again
• multiline is difficult
because data from
containers can be mixed
@monicasarbu
33
100 Journald driver + Filebeat
Pros:
Cons:
• journald is often already
available
• Filebeat doesn’t yet
support journald
• convenient support for
container metadata
(name, labels, etc.)
• You can use the
community Beat,
Journalbeat
• `docker logs` works
@monicasarbu
34
101 Shared volume + Filebeat
Pros:
Cons:
• If your app can rotate it’s
own logs, it’s very easy to
setup
• Difficult to pass container
metadata (name, labels,
etc.)
• Scales well
@monicasarbu
35
Conclusion
“At-least-once” guarantees and handle
back-pressure:
• json-file driver + Filebeat
No guarantees:
• Gelf driver + Logstash
• Fluentd driver + Logstash
• Syslog driver + Filebeat
• Shared volume + Filebeat
• Journald driver + Filebeat (in the
future)
@monicasarbu
36
Metricbeat
new in 5.0
37
One Metricbeat module for each service
+
Add your own
@monicasarbu
38
Metricbeat system module
@monicasarbu
CPU
Mem
diskIO
filesystem
load
network
cores
processes
39
Metricbeat
Collect container metrics
40
Querying the Docker API
• CPU and memory
• Docker container information
• network (in/out bytes, dropped)
• diskIO (reads/writes)
• status of containers (# of stopped, running, etc)
@monicasarbu
41
Docker module in Metricbeat
in progress
• Get container metrics by querying the Docker API
• Has access to container names and labels
• Easy to setup
@monicasarbu
42
Reading cgroup data from /proc/
• Doesn’t require access to the Docker API
(can be a security issue)
• Works for any container runtime (Docker,
rkt, runC, LXD, etc.)
• Cannot get the container name and labels
only the container ID
@monicasarbu
43
System module + cgroup data
• if cgroup option is enabled (by default is
disabled)
• Automatically enhances process data with
cgroup information
@monicasarbu
44
Run as a container
App1
App2
App3
Host
@monicasarbu
45
Elasticsearch as time series DB
46
Elasticsearch BKD trees
• Added for Geo-points
• faster to index
#velo
• faster to query
• more disk-efficient
• more memory efficient
@monicasarbu
47
Float values
OnDiskUsageinkb
80000
• half floats
70000
60000
• scaled floats (using a
scaling factor) - great for
things like percentage
points
50000
40000
30000
20000
10000
0
float
halffloat
Pointsdisk usage(kb)
@monicasarbu
scaledfloat
(factor=4000)
scaledfloat
(factor=100)
docs_values disk usage(kb)
48
Why Elasticsearch for time series
• Horizontal scalability. Mature and
battle tested cluster support.
• Flexible aggregations (incl moving
averages & Holt Winters)
#velo
• One system for both logs and
metrics
• Timelion UI, Grafana
• Great ecosystem: e.g. alerting
tools
@monicasarbu
49
Packetbeat
50
How Packetbeat works
@monicasarbu
1
2
capture
network traffic
decodes
network traffic
3
correlates
request &
response into
transactions
4
send
transactions to
Elasticsearch
51
Supported traffic decoders
http://
Thrift
ICMP
DNS
AMQP
+
Add your own
@monicasarbu
52
Unknown traffic, use flows
•Look into data for which we don’t
understand the application layer protocol
•TLS
•Protocols we don’t yet support
•Get data about IP / TCP / UDP layers
•number of packets & bytes
•retransmissions
•inter-arrival time
@monicasarbu
53
Monitor traffic exchanged by containers
App1
App2
App3
Packetbeat
Host
traffic exchanged between your containers
@monicasarbu
54
Demo: Metricbeat, Filebeat,
Packetbeat
Multiple data types, one view in Kibana
55
Thank you
• github.com/elastic/beats
• discuss.elastic.co
• @elastic #elasticbeats
• #beats on freenode
56