IBM Security QRadar August 24, 2015
QRadar Open Mic #10: Let's talk
about Log Source Extensions
Panelists
Adam Frank – Principal Solutions Architect
Alaa Ali – Security Consultant, Lab Services Team
Colin Hay – Team Lead, QRadar Ecosystem and Integrations
Dwight Spencer – Principal Solutions Architect & Co-founder of Q1 Labs
Jeff Rusk – Development Manager, QRadar Integration Services
Jonathan Pechta – Support Technical Writer
Mark Wright – QRadar L2 Support Manager
Reminder: You must dial-in to the phone conference to listen
to the panelists. The web cast does not include audio.
• USA toll-free: 1-866-803-2145
• Participant passcode: 4105322
• Slides and additional dial in numbers: https://ibm.biz/BdX3tQ
NOTICE: By participating in this call, you give your irrevocable consent to IBM
to record any statements that you may make during the call, as well as to
IBM's use of such recording in any and all media, including for video postings
on YouTube. If you object, please do not connect to this call.
© 2015 IBM Corporation
Table of contents / Agenda
 About log source extensions
 When to use log source extensions
 Getting your example payloads
 Structure of a log source extension
 Identity
 Limitations and tips
 Creating QID map entries (qidmap_cli.sh)
 Uploading and assigning your LSX
 Mapping events
 Discussion & questions
© 2015 IBM Corporation
2
About Log Source Extensions
© 2015 IBM Corporation
About Log Source Extensions
Log Source Extension (LSXs) when applied to a DSM provide the logic for parsing
normalized/common event data. The LSX can be applied to either a Universal DSM (UDSM) or
an officially support DSM to parse events from data being sent to QRadar. The most common
use of LSXs is to integrate events for log sources that require a UDSM, however, they can also
be used to override parsing for existing DSMs.
LSXs are based on Java regular expressions and can be used to parse events for any protocol
supported by QRadar. For example, Syslog, JDBC, Log File protocol,, etc.
LSXs Values can be extracted from the logs and mapped to all common fields within QRadar.
EventName
EventCategory
SourceIp
SourcePort
SourceIpPreNAT
SourceIpPostNAT
SourceMAC
SourcePortPreNAT
SourcePortPostNAT
DestinationIp
DestinationPort
DestinationIpPreNAT
DestinationIpPostNAT
DestinationPortPreNAT
DestinationPortPostNAT
DestinationMAC
DeviceTime
Protocol
UserName
HostName
GroupName
NetBIOSName
ExtraIdentityData
SourceIpv6
DestinationIpv6
© 2015 IBM Corporation
4
When to use a Log Source
Extensions
© 2015 IBM Corporation
When do I need to use a Log Source Extension?
Sometimes it can be difficult to decide if you really need a log source extension and UDSM or if
you can get by with just creating a custom event property for the data you want to extract.
If you are just looking for one or two specific pieces of information from an event, then you can
save time by just creating a custom event property. For example, I have a UPS system that is
does not have a supported event. In my case, all I care about is two pieces of data: 1. failures
and 2. battery messages. I can just create a custom event property to find that data and create
offenses around those optimized properties.
However, if I have a device that generates multiple different events, then it is much more
efficient to use a UDSM and log source extension to parse out the different data and map them
to events.
© 2015 IBM Corporation
6
Getting your example payloads
© 2015 IBM Corporation
Getting your example payloads
Before administrators can start to write extensions, it is
important to understand the data format for the event.
If possible, customers can always:
1. Get logs from the device owner.
2. Forward data to QRadar.
1.
2.
3.
4.
5.
6.
7.
8.
9.
From the Admin tab, create a new source
by clicking the Log Sources icon.
Click Add.
Specify a name in the Log Source Name field.
From the Log Source Type list, select
Universal DSM.
From the Protocol Configuration, select a
protocol.
Tip: Disable the Coalescing check
For the Log Source Identifier, enter either
box when you first start to ensure that
the IP address or host name of the event source.
you are seeing all of the event data.
Click Save.
From the Admin tab, click Deploy Changes.
This data can then be exported from the Log Activity tab.
© 2015 IBM Corporation
8
Compare the example payloads to the extension template
Example 1:
May 20 17:16:14 hostname[22331]: bad password attempt for 'root‘ from 192.168.50.80:80
Example 2:
<13>Aug 24 06:51:23 10.10.10.10 AgentDevice=WindowsLog AgentLogFile=Security Source=MicrosoftWindows-Security-Auditing Computer=WIN-8VM.lab User= Domain= EventID=4634 EventIDCode=4634
EventType=8 EventCategory=12545 RecordNumber=3838837 TimeGenerated=1440409314934
TimeWritten=1440409314934 Message=An account was logged off. Subject: Security ID: S-1-5-18 Account
Name: WIN-8VMAccount Domain: LAB Logon ID: 0x2a111c111 Logon Type: 3
Example 3:
<142>Mar 12 17:13:10 apache.httpserver.test.com httpd[1402]: 172.16.77.1 10.10.10.10 - [12/Mar/2013:17:13:10 +0000] "GET /maint/default/maintenance.html HTTP/1.1" 200 80 1658
EventName
EventCategory
SourceIp
SourcePort
SourceIpPreNAT
SourceIpPostNAT
SourceMAC
SourcePortPreNAT
SourcePortPostNAT
DestinationIp
DestinationPort
DestinationIpPreNAT
DestinationIpPostNAT
DestinationPortPreNAT
DestinationPortPostNAT
DestinationMAC
DeviceTime
Protocol
UserName
HostName
GroupName
NetBIOSName
ExtraIdentityData
SourceIpv6
DestinationIpv6
© 2015 IBM Corporation
9
Structure of an Log Source
Extension
© 2015 IBM Corporation
Structure of a log source extension
© 2015 IBM Corporation
11
Structure of a log source extension
 Pattern: A pattern is the area where a name/identifier is declared and associated to the Java
regular expression.
CDATA: All characters between the start tag <pattern> and end tag </pattern> are
considered part of the pattern. Do not use extra spaces or hard returns inside or around your
pattern or <CDATA> expression. Extra characters or spaces can prevent the DSM extension
from matching your intended pattern
In patterns, there are options for defining case sensitivity and trimming white space.
Capture groups: Multiple sets of (regex1)(regex2)(regex3) can be define as capture groups
for the pattern.
<pattern id="SourceIp" xmlns=""><![CDATA[addr (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})]]></pattern>
Capture groups are very useful when you have combination data in a single field that needs
to be extracted to individual categories. For example 10.10.10.10:8080.
<! [CDATA[Source=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}):([\d]{1,5})]]></pattern>
<matcher field="SourceIp" order="1" pattern-id="SourceIPColonPort" capturegroup="1"/>
<matcher field="SourcePort" order="1" pattern-id="SourceIPColonPort" capturegroup="2"/>
© 2015 IBM Corporation
12
Structure of a log source extension
 Matcher: A matcher defines the field being parsed and adds options for parsing, such as
order, capture groups, enabling substitutions, and extra data (ext-data). For example,
“SourceIp” shared a pattern-id, which allows the two fields to be associated.
<matcher field="SourceIp" order="1" pattern-id="SourceIp" capture-group="1" />
Options: The 3rd section for the event matcher allows the user to define special parameters
on the matcher group, such as identify flags, defining the device category, and severity.
<event-match-single event-name=“AuthEvent" device-event-category=“LinuxOS”
severity="6" send-identity=“OverrideAndNeverSend"/>
© 2015 IBM Corporation
13
Identity
© 2015 IBM Corporation
What is identity?
Identity information allows QRadar to provide IP address, hostname, MAC address, or NetBIOS
information to the asset profiler for asset reconciliation. There are several values that can be
used for sending identity, but most users will want to never send identity information. The
values are OverrideAndNeverSend, OverrideAndAlwaysSend, UseDSMResult, SendifAbsent.
send-identity=“OverrideAndNeverSend"
Asset reconciliation is how QRadar updates assets with new data from events when identity
information is provided. A basic example of asset reconciliation:
1. Examine the data for identifiers in the update to the database of existing assets and find a
match.
2. If the update contained a known MAC, NetBIOS, or DNS Hostname, then update values
and populate any new data or new information that can be provided.
3. If the matching value is an IP Address, a match to an asset is based on the other
information in the update versus what is already known about the potential IP-matching
asset in the database.
4. In the case of no matching data from 2 or 3, a brand new asset is created to accommodate
the information provided in the asset update.
© 2015 IBM Corporation
15
Limitations and tips
© 2015 IBM Corporation
Limitations of a log source extension
 A log source extension is unable to 'change' or substitute the event name of an event. The
event name is one field that cannot be changed. The matched field can be mapped to
another attribute, but the actual event name cannot be changed.
 Log source extensions cannot properly process multiline events. QRadar is designed to
interpret single line events.
However, there are protocols that can read multiline events, such as the TCP Multiline
Syslog protocol or the UDP Multiline Syslog protocol for streaming events. If you are using
the Log File Protocol, there are event generators that can be used to process multiline
events from a flat file, such as ID-Linked Multiline or Regex Based Multiline.
If none of these options work for you, then your multiline logs may require some preprocessing make the files single line format, which can then be parsed by the log source
extension. For example using a tool such as xml2csv for making the multiple lines into a
single easy to parse line.
 You cannot use variables, such as $1 or $2 in a log source extension.
 LSXs can be either US-ASCII or UTF-8. If another format is required, it must be defined in
the extension <device-extension source-encoding=SHIFT-JIS xmlns=event_parsing/device_extension>
© 2015 IBM Corporation
17
Log Source Time
 Log Source Time in the user interface equates to DeviceTime in your extension.
<matcher field="DeviceTime" order="1" ext-data="MMM d HH:mm:ss"
pattern-id="DeviceTime" capture-group="1" />
 Start Time is the time that the event entered the event (ECS) pipeline on the QRadar
appliance. This value is set by QRadar and cannot be defined in the LSX.
 Storage Time is the time that the event was written to disk on the QRadar appliance. This
value is set by QRadar and cannot be defined in the LSX.
© 2015 IBM Corporation
18
What is the purpose of allEventNames?
<!-- Do not remove the "allEventNames" value -->
<pattern id="allEventNames" xmlns=""><![CDATA[(.*)]]></pattern>
 The purpose of the allEventNames pattern is to ensure that the entire event payload is
captured in QRadar.
Without this line at the top, the LSX might not forward all of the data for the event payload
and only forward the pieces matched in the regular expressions you define. So, with the
allEventNames line at the top, you can ensure that the entire event payload can be viewed
for your unknown events and the sub-entries an further refine the information you are trying
to parse to create associations to the UI fields. The allEventNames pattern isn't required,
depending on how you are writing your LSX, but I think it is probably rarely to not have it
because that line can ensure that you are getting the entire picture for your event payload
© 2015 IBM Corporation
19
What is the difference between a parsing Overrides &
Enhancement?
 None, there is no difference between the two. All extensions act as overrides for the data
defined within the log source extension.
 As you know, a log source extension can be used to parse data from an unknown or
unsupported device, but it can also be used to customize or update the parsing for an official
DSM created by the IBM Integration Team. The difference comes from how you write the
LSX as you can write your LSX.
The LSX might contain a single event pattern and matcher to resolve as specific issue in for
your device. This allows administrators to deal with parsing issues while waiting for support
for a specific product version.
© 2015 IBM Corporation
20
About Syslog Events and QRadar
 QRadar officially supports syslog RFC3164 and RFC5424 from most event sources that are
listed as "Syslog" in the documentation.
If a specialized or vendor version of Syslog is supported, such as LEEF for QRadar or CEF,
then the documentation will define the format as LEEF or CEF. For QRadar 7.1 customers,
RFC5424 support was added in QRadar 7.1 MR2 Patch 7 and above. RFC5424 support was
documented as a closed issue in APAR IV61716.
 QRadar evaluates IP addresses from Syslog events using the following priority:
1.
2.
3.
Packet IP address (highest priority)
Syslog header IP address
Syslog payload IP address (lowest priority)
© 2015 IBM Corporation
21
Creating custom QIDs
© 2015 IBM Corporation
What is a QID?
 A QID is a “QRadar Identifier” and is used to map an event to a description. QID identifies
following information in the user interface for an event:
–
–
–
–
Event Name
Event Category
Event severity
Event description
qidmap_cli.sh -c --qname <name> --qdescription <description> -severity <severity> --lowlevelcategoryid <ID>
 Administrators can define their own QIDs in QRadar, then map their events to the custom
QIDs. Creating custom QIDs is done via the command-line with the
opt/qradar/bin/qidmap_cli.sh script. The script can create or modify single entries or import a
list of QIDs from a CSV file.
 Custom QIDs start at number 2,000,000 and end at 2,250,000.
© 2015 IBM Corporation
23
Where do I find a list of low level categories?
 Customers can list the low level categories in QRadar by using the –l command.
It is also helpful when setting up extensions to export the list of low level categories to a file.
This enables users to start a spreadsheet with a column for name, description, severity file.
qidmap_cli.sh -l > lowlevelcats.txt
TIP: If you are doing a large number of QIDs, it might be useful to keep a spreadsheet of your progress.
QIDs when imported are part of the QRadar database, so configuration backups keep your data. However,
if someone needs to review or do updates, a spreadsheet is a good way to hand off work to others.
© 2015 IBM Corporation
24
Uploading and assigning your LSX
© 2015 IBM Corporation
Uploading your log source extension
 To upload your extension, you must be an administrator:
Admin tab > Log Source Extensions > Browse > Upload
© 2015 IBM Corporation
26
Assigning the log source extension to a log source
 To select an extension for your log source, you must be an administrator.
Admin tab > Log Sources > Add or Edit (double-click) > Select your
Log Source Extension
© 2015 IBM Corporation
27
Mapping events
© 2015 IBM Corporation
Mapping events
 Any user with the role of permission to the Log Activity tab and the
 To map an event: Log Activity tab > Double-click an event > Map Event
© 2015 IBM Corporation
29
Mapping events
 When mapping events, the user
can search by name, QID number,
or any of the other values listed on
screen.
 Run a search against your Log
Source to verify the event is
mapped properly and all of the
data is correct in the log source
extension.
© 2015 IBM Corporation
30
Questions & discussion
© 2015 IBM Corporation
Questions for the panel?
Now is your opportunity to ask questions of our panelists.
To ask a question now:
Press
*1 to ask a question over the phone
or
Type your question into the SmartCloud Meetings chat
To ask a question after this presentation:
You are encouraged to participate in our Forum topics <link to IBM Security topic A
in product X’s Forum>.
© 2015 IBM Corporation
32
Statement of Good Security Practices: IT system security involves protecting systems and information through prevention, detection and response to improper access from within and outside
your enterprise. Improper access can result in information being altered, destroyed, misappropriated or misused or can result in damage to or misuse of your systems, including for use in attacks
on others. No IT system or product should be considered completely secure and no single product, service or security measure can be completely effective in preventing improper use or access.
IBM systems, products and services are designed to be part of a lawful, comprehensive security approach, which will necessarily involve additional operational procedures, and may require other
systems, products or services to be most effective. IBM DOES NOT WARRANT THAT ANY SYSTEMS, PRODUCTS OR SERVICES ARE IMMUNE FROM, OR WILL MAKE YOUR ENTERPRISE
IMMUNE FROM, THE MALICIOUS OR ILLEGAL CONDUCT OF ANY PARTY.
THANK YOU
www.ibm.com/security
© Copyright IBM Corporation 2015. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any
kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor
shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use
of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and / or
capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product
or feature availability in any way. IBM, the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries
or both. Other company, product, or service names may be trademarks or service marks of others.