OWASP ModSecurity Core Rule
Set (CRS) Project
Ryan Barnett
OWASP CRS Project Leader
Senior Security Researcher
Ryan Barnett Background
•Trustwave
•SpiderLabs Research Team
•Web application firewall
research/development
•ModSecurity Community Manager
•Interface with the community on
public mail-list
•Steer the internal development of
ModSecurity
•Author
•“Preventing Web Attacks with Apache”
Community Projects
• Open Web Application Security Project (OWASP)
• Project Leader, ModSecurity Core Rule Set
• Project Contributor, OWASP Top 10
• Project Contributor, AppSensor
• Web Application Security Consortium (WASC)
• Project Leader, Web Hacking Incident Database
• Project Leader, Distributed Web Honeypots
• Project Contributor, Web Application Firewall Evaluation Criteria
• Project Contributor, Threat Classification
• The SANS Institute
• Courseware Developer/Instructor
• Project Contributor, CWE/SANS Top 25 Worst Programming Errors
Session Outline
• ModSecurity Quick Overview
• The Core Rule Set (CRS) Overview
• Basic Detection Categories
• Latest CRS Improvements
• CRS Demonstration Page
• Future Directions
ModSecurity Quick Overview
What is ModSecurity?
 It is an open source web application firewall (WAF)
module for Apache web servers
www.modsecurity.org
Separate Rule and Audit Engines
Allows full request/response HTTP logging capability
Deep understanding of HTTP and HTML
Robust Parsing (form encoding, multipart, XML)
Event-based Rules Language
Anti-Evasion Features (normalization functions)
Advanced Capabilities
Transactional and Persistent Collections
Content Injection
Lua API
ModSecurity’s Rules Language Syntax
Tells ModSecurity how
to process data (such
@rx, @pm or @gt).
SecRule TARGETS OPERATOR [ACTIONS]
Tells ModSecurity where
to look (such as ARGS,
ARGS_NAMES or
COOKIES).
Tells ModSecurity what to
do if a rule matches (such
as deny, exec or setvar).
ModSecurity’s Apache Request Cycle Hooks
OWASP ModSecurity Core Rule
Set (CRS) Overview
Project Info
http://www.owasp.org/index.php/Category:OWASP_ModSecurity_Core_Rule_Set_Project
What is the Core Rule Set (CRS)?
A generic, plug-n-play set of WAF rules
Choose your mode of operation
Standard vs. Anomaly Scoring
Detection Categories:
Protocol Validation
Malicious Client Identification
Generic Attack Signatures
Known Vulnerabilities Signatures
Trojan/Backdoor Access
Outbound Data Leakage
Anti-Virus and DoS utility scripts
Initial Configuration
Once unpacked, edit the main config file
modsecurity_crs_10_config.conf
Customize the following items
Mode of Detection – Standard vs. Anomaly Scoring
Anomaly Scoring Severity Levels
Enable/Disable Blocking
Blocking Threshold Levels
Paranoid Mode – aggressive inspection
HTTP Policy Settings
Choose where to log events (Apache error_log and/or
ModSecurity’s audit log)
Traditional Detection Mode
Self Contained Rules Concept
IDS/IPS mode with “self-contained” rules
Like HTTP itself – the rules are stateless
No intelligence is shared between rules
If a rule triggers, it will execute a disruptive/logging action
Easier for the new user to understand
Not optimal from a rules management perspective
(handling false positives/exceptions)
Not optimal from a security perspective
Not every site has the same risk tolerance
Lower severity alerts are largely ignored
Anomaly Scoring Detection Mode
Collaborative Rules Concept
Advanced inspection/detection mode
Delayed blocking
Rules set transactional variables (tx) to store temporary
meta-data about the rule match
Rules also increase anomaly scores for both the attack
category and global score
The anomaly score enforcement rules decide whether
or not to deny transactions at the end of the inbound
request phase
modsecurity_crs_49_inbound_blocking.conf
Anomaly Scoring - Debug Log View
Target value: "" OR 1=1#"
AdResolved macro %{rule.msg} to: SQL Injection AttackSet variable "tx.msg" to "SQL Injection Attack".
Setting variable: tx.sql_injection_score=+%{tx.critical_anomaly_score}
Recorded original collection variable: tx.sql_injection_score = "0"
Resolved macro %{tx.critical_anomaly_score} to: 5
Relative change: sql_injection_score=0+5
Set variable "tx.sql_injection_score" to "5".
Setting variable: tx.anomaly_score=+%{tx.critical_anomaly_score}
Original collection variable: tx.anomaly_score = "3"
Resolved macro %{tx.critical_anomaly_score} to: 5
Relative change: anomaly_score=3+5
Set variable "tx.anomaly_score" to "8".
Setting variable: tx.%{rule.id}-WEB_ATTACK/SQL_INJECTION-%{matched_var_name}=%{tx.0}
Resolved macro %{rule.id} to: 950901
Resolved macro %{matched_var_name} to: ARGS:prod_id
Resolved macro %{tx.0} to: 1=1
Set variable "tx.950901-WEB_ATTACK/SQL_INJECTION-ARGS:prod_id" to "1=1".
Warning. Pattern match "\b(\d+) ?(?:=|<>|<=>|<|>|!=) ?\1\b|[\'"\`\´\’\‘](\d+)[\'"\`\´\’\‘]
?(?:=|<>|<=>|<|>|!=) ?[\'"\`\´\’\‘]\2\b|[\'"\`\´\‘](\w+)[\'"\`\´\’\‘] ?(?:=|<>|<=>|<|>|!=)
?[\'"\`\´\’\‘]\3\b|([\'"\;\`\´\’\‘]*)?\s+(and|or)\s+([\s\'"\` ..." at ARGS:prod_id. [file
"/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_41_sql_injection_attacks.conf"]
[line "425"] [id "950901"] [rev "2.0.9"] [msg "SQL Injection Attack"] [data "1=1"] [severity
"CRITICAL"]
ded regex subexpression to TX.0: 1=1Added regex subexpression to TX.1: 1
Operator completed in 19 usec.Ctl: Set auditLogParts to ABIFHZE.Setting variable: tx.msg=%{rule.msg}
Inspecting Anomaly Scores
# Alert and Block based on Anomaly Scores
#
SecRule TX:ANOMALY_SCORE "@gt 0" \
"chain,phase:2,t:none,nolog,auditlog,deny,msg:'Inbound Anomaly
Score Exceeded (Total Score: %{TX.ANOMALY_SCORE},
SQLi=%{TX.SQL_INJECTION_SCORE}, XSS=%{TX.XSS_SCORE}):
%{tx.msg}',setvar:tx.inbound_tx_msg=%{tx.msg},setvar:tx.inbound_anom
aly_score=%{tx.anomaly_score}"
SecRule TX:ANOMALY_SCORE "@ge
%{tx.inbound_anomaly_score_level}" chain
SecRule TX:ANOMALY_SCORE_BLOCKING "@streq on"
Conditional Rules (Weak Sigs)
SQL Injection Example
Aggregate indicators to determine an attack
Strong indicators
Keywords such as: xp_cmdshell, varchar,
Sequences such as: union …. select, select … top … 1
Amount: script, cookie and document appear in the
same input field
Weak indicators – meta-characters
--, ;, ', …
CRS only applies weak signatures in the event a
stronger signature has previously triggered
Conditional Rule Example
SecMarker BEGIN_SQL_INJECTION_WEAK
SecRule &TX:/SQL_INJECTION/ "@eq 0"
"phase:2,t:none,nolog,pass,skipAfter:END_SQL_INJECTION_WEAK"
SecRule TX:/SQL_INJECTION/
"\b(?:rel(?:(?:nam|typ)e|kind)|a(?:ttn(?:ame|um)|scii)|c(?:o(?:nver|un)t|ha?r
)|s(?:hutdown|elect)|to_(?:numbe|cha)r|u(?:pdate|nion)|d(?:elete|rop)|group\b
\W*\bby|having|insert|length|where)\b" \
"phase:2,chain,capture,t:none,ctl:auditLogParts=+E,block,nolog,auditlog
,msg:'SQL Injection
Attack',id:'950001',tag:'WEB_ATTACK/SQL_INJECTION',logdata:'%{TX.0}',severity
:'2'"
SecRule MATCHED_VAR "(?:[\\\(\)\%#]|--)" \
"t:none,setvar:'tx.msg=%{rule.msg}',setvar:tx.sqli_score=+1,setvar:tx.a
nomaly_score=+20,setvar:tx.%{rule.id}-WEB_ATTACK/SQL_INJECTION%{matched_var_name}=%{matched_var}"
SecMarker END_SQL_INJECTION_WEAK
Event Logging
Standard vs. Correlated Events
Standard mode
Rules log event data to both the Apache error_log and the
ModSecurity Audit log
Correlated mode
Basic rules are considered reference events and do not
directly log to the Apache error_log
Correlation rules in the logging phase analyze
inbound/outbound events and generate special events
modsecurity_crs_60_correlation.conf
Inbound/Outbound Correlation
Couple the inbound with the outbound for increased
intelligence
Was there an inbound attack?
Was there an HTTP Status Code Error (4xx/5xx level)?
Was there an application information leak?
Correlation facilitates better incident response
App error without inbound attack -> Contact Ops
Inbound attack + outbound error -> Contact Security
Event Severity Ratings
 Correlated Events
 0: Emergency - is generated from correlation (inbound attack +
outbound leakage)
 1: Alert - is generated from correlation (inbound attack + outbound
application level error)
 Non-Correlated Events
 2: Critical - highest severity level possible without correlation. It is
normally generated by the web attack rules (40 level files)
 3: Error - is generated from outbound leakage rules (50 level files)
 4: Warning - is generated by malicious client rules (35 level files)
 5: Notice - is generated by the Protocol policy and anomaly files
 6: Info - is generated by the search engine clients (55 marketing file)
Correlated Event Messages
Message: Pattern match "\;\W*?\bdrop\b" at TX:pm_sqli_data_REQUEST_URI.
[file "/opt/waschoneypot/etc/rules/base_rules/modsecurity_crs_41_sql_injection_attacks.conf"
] [line "262"] [id "959001"] [msg "SQL Injection Attack"] [data "; drop"]
[severity "CRITICAL"] [tag "WEB_ATTACK/SQL_INJECTION"]
Message: Operator GE matched 0 at TX:anomaly_score. [file "/opt/waschoneypot/etc/rules/base_rules/modsecurity_crs_49_enforcement.conf"] [line
"30"] [msg "Anomaly Score Exceeded (score 55): SQL Injection Attack
Detected"]
Message: Pattern match "\bsupplied argument is not a valid MySQL\b" at
RESPONSE_BODY. [file "/opt/waschoneypot/etc/rules/base_rules/modsecurity_crs_50_outbound.conf"] [line
"259"] [id "971156"] [msg "SQL Information Leakage"] [severity "ERROR"] [tag
"LEAKAGE/ERRORS"]
Message: Warning. Operator GE matched 1 at TX. [file "/opt/waschoneypot/etc/rules/base_rules/modsecurity_crs_60_correlation.conf"] [line
"24"] [msg "Correlated Successful Attack Identified: Inbound Attack (SQL
Injection Attack Detected) + Outbound Data Leakage (SQL Information Leakage)
- (Transactional Anomaly Score: 85)"] [severity "EMERGENCY"]
Detection Mechanisms: Protocol Violations
Protocol vulnerabilities such as Response Splitting,
Request Smuggling, Premature URL ending
Content length only for non GET/HEAD methods
Non ASCII characters or encoding in headers
Valid use of headers (for example, content length is numerical)
Proxy Access
modsecurity_crs_20_protocol_violations.conf
Attack requests are different due to automation
Missing headers such as Host, Accept, User-Agent
Host is an IP address (common worm propagation method)
modsecurity_crs_21_protocol_anomalies.conf
Detection Mechanisms: Protocol Policies
Policy is usually application specific
Some restrictions can usually be applied generically
White lists can be build for specific environments
Limitations on Sizes
Request size, Upload size
# of parameters, length of parameter
modsecurity_crs_23_request_limits.conf
Items that can be allowed or restricted
Methods - Allow or restrict WebDAV, block abused methods
such as CONNECT, TRACE or DEBUG
File extensions – backup files, database files, ini files
Content-Types (and to some extent other headers)
Modsecurity_crs_30_http_policy.conf
Detection Mechanisms: Malicious Clients
Not aimed against targeted attacks, but against general
malicious internet activity
Offloads a lot of cyberspace junk & noise
Effective against comment spam
Reduce event count
Detection of Malicious Robots
Unique request attributes: User-Agent header, URL, Headers
Black list of IP addresses
Rate based detection
Detection of security scanners
Blocking can confuse security testing software (WAFW00f)
 modsecurity_crs_35_bad_robots.conf
Detection Mechanisms: App Layer Attacks
Detect application level attacks such as those described
in the OWASP top 10
SQL injection and blind SQL injection
Cross site scripting (XSS)
OS command injection and remote command access
Remote file inclusion
 modsecurity_crs_40_generic_attacks.conf
 modsecurity_crs_41_sql_injection_attacks.conf
 modsecurity_crs_41_xss_attacks.conf
Known Vulnerability Signatures
 SpiderLabs received authorization from ET to convert
their Snort rules and include them in the CRS
 http://www.emergingthreats.net/
 Converted the following rule files
 emerging-web_server.rules
 emerging-web_specific_apps.rules
 Identifying attacks against known vulnerabilities does
have value
 Raised threat level
 If done correctly, lessens false positives
 CRS combines the what of our generic attack payload
detection with the where of ET known vuln data
Example Emerging Threats Rule
Attack vectorany -> $HTTP_SERVERS
alert tcp $EXTERNAL_NET
location –
$HTTP_PORTS (msg:"ET
WEB_SPECIFIC_APPS 20/20 Auto
URI + Parameter
Gallery SQL Injection Attempt -- vehiclelistings.asp
vehicleID SELECT"; flow:established,to_server;
uricontent:"/vehiclelistings.asp?"; nocase;
uricontent:"vehicleID="; nocase; uricontent:"SELECT";
nocase; pcre:"/.+SELECT.+FROM/Ui"; classtype:webapplication-attack; reference:cve,CVE-2006-6092;
reference:url,www.securityfocus.com/bid/21154;
reference:url,doc.emergingthreats.net/2007504;
PCRE –
reference:url,www.emergingthreats.net/cgiWeak signature
bin/cvsweb.cgi/sigs/WEB_SPECIFIC_APPS/WEB_2020_Auto_g
allery; sid:2007504; rev:5;)
Converted Emerging Threats Rule
Verify the URI of
the request
# (sid 2007508) ET WEB_SPECIFIC 20/20 Auto Gallery SQL Injection
Attempt -- vehiclelistings.asp vehicleID
SecRule REQUEST_URI_RAW "(?i:\/vehiclelistings\.asp)"
"chain,phase:2,block,t:none,t:urlDecodeUni,t:htmlEntityDecode,t:norma
lisePathWin,capture,ctl:auditLogParts=+E,nolog,auditlog,logdata:'%{TX
.0}',id:sid2007508,rev:3,msg:'ET WEB_SPECIFIC 20/20 Auto Gallery SQL
Verify the attack vector
Injection Attempt -- vehiclelistings.asp
vehicleID ',tag:‘weblocation from saved TX SQL
application-attack',tag:'url,www.emergingthreats.net/cgiInjection data exists
bin/cvsweb.cgi/sigs/WEB_SQL_INJECTION/WEB_2020_Auto_gallery'"
SecRule &TX:'/SQL_INJECTION.*ARGS:vehicleID/' "@gt 0"
"setvar:'tx.msg=ET WEB_SPECIFIC 20/20 Auto Gallery SQL Injection
Attempt -- vehiclelistings.asp vehicleID
',setvar:tx.sqli_score=+1,setvar:tx.anomaly_score=+20,setvar:tx.%{rul
e.id}-SQL_INJECTION/SQL_INJECTION-%{matched_var_name}=%{matched_var}"
Detection Mechanisms: Trojans/Backdoors
Major problem in hosting environments
Uploading is allowed
Some sites may be secure while others not
Upload detection
Check uploading of files containing viruses (i.e. WORD docs)
util/modsec-clamscan.pl
Check uploading of http backdoor page
Access detection
Known signatures (x_key header)
Generic file management output (gid, uid, drwx, c:\)
 modsecurity_crs_45_trojans.conf
Detection Mechanisms: Information
Leakage
Monitoring outbound application data
HTTP Error Response Status Codes
SQL Information Leakage
Stack Dumps
Source Code Leakage
Last line of defense if all else fails
Provide feedback to application developers
Important for customer experience
Makes life for the hacker harder (if blocking is used)
modsecurity_crs_50_outbound.conf
Latest CRS Improvements:
v2.0.9
Latest Improvements
Lua port of PHPIDS Converter.php code
Advanced normalization functions
More accurate use of PHPIDS Filters
Centrifuge Generic Attack Payload Detection
Experimental Rules
Generic attack payload detection
CRS Demonstration Page
Request Header Tagging
Lua port of PHPIDS
 http://phpids.net/
 ~70 regular expression rules to detect common attack
payloads
 XSS
 SQL Injection
 RFI
 Filters are heavily tested by the community and updated
frequently
 Trustwave SpiderLabs worked with PHPIDS lead to port
code to Lua for use in ModSecurity’s API
 https://svn.php-ids.org/svn/trunk/lib/IDS/Converter.php
 https://svn.php-ids.org/svn/trunk/lib/IDS/default_filter.xml
 Thanks to Mario Heiderich
Lua port of PHPIDS
 Example normalization functions
 --[[ Make sure the value to normalize and monitor doesn't contain Regex DoS
 --[[ Check for comments and erases them if available ]]
 --[[ Strip newlines ]]
 --[[ Checks for common charcode pattern and decodes them ]]
 --[[ Eliminate JS regex modifiers ]]
 --[[ Converts from hex/dec entities ]]
 --[[ Normalize Quotes ]]
 --[[ Converts SQLHEX to plain text ]]
 --[[ Converts basic SQL keywords and obfuscations ]]
 --[[ Detects nullbytes and controls chars via ord() ]]
 --[[ This method matches and translates base64 strings and fragments ]]
 --[[ Strip XML patterns ]]
 --[[ This method converts JS unicode code points to regular characters ]]
 --[[ Converts relevant UTF-7 tags to UTF-8 ]]
 --[[ Converts basic concatenations ]]
 --[[ This method collects and decodes proprietary encoding types ]]
PHPIDS Example Filter
<filter>
<id>1</id>
<rule><![CDATA[(?:"[^"]*[^]?>)|(?:[^\w\s]\s*\/>)|(?:>")]]></rule>
<description>finds html breaking injections
including whitespace attacks</description>
<tags>
<tag>xss</tag>
<tag>csrf</tag>
</tags>
<impact>4</impact>
</filter>
Converted PHPIDS Example Filters
SecRule TX:'/^(QUERY_|REQUEST_|ARGS:).*_normalized/'
"(?:\<\w*:?\s(?:[^\>]*)t(?!rong))|(?:\<scri)|(<\w+:\w+)"
"phase:2,capture,t:none,pass,skip:1,nolog,auditlog,msg:'Detects
obfuscated script tags and XML wrapped
HTML',id:'9000033',tag:'WEB_ATTACK/XSS',logdata:'%{TX.0}',severity:
'2',setvar:'tx.msg=%{rule.id}%{rule.msg}',setvar:tx.anomaly_score=+4,setvar:'tx.%{tx.msg}WEB_ATTACK/XSS-%{matched_var_name}=%{tx.0}'"
SecRule TX:PARANOID_MODE "@eq 1"
"chain,phase:2,t:none,logdata:'%{TX.0}',severity:'2',pass,nolog,aud
itlog,msg:'Detects obfuscated script tags and XML wrapped
HTML',id:'9000033',tag:'WEB_ATTACK/XSS'"
SecRule ARGS|REQUEST_BODY|REQUEST_URI_RAW
"(?:\<\w*:?\s(?:[^\>]*)t(?!rong))|(?:\<scri)|(<\w+:\w+)"
"capture,multiMatch,t:none,t:urlDecodeUni,t:cssDecode,t:jsDecode,t:
htmlEntityDecode,t:replaceComments,t:compressWhiteSpace,t:lowercase
,setvar:'tx.msg=%{rule.id}%{rule.msg}',setvar:tx.anomaly_score=+4,setvar:'tx.%{tx.msg}WEB_ATTACK/XSS-%{matched_var_name}=%{tx.0}'"
Centrifuge Code
 Negative security approach to combating XSS and SQL
Injection is doomed to fail…
 Unlimited ways to write functionally equivalent code
 Obfuscation methods, however often have certain characteristics
 PHPIDS has an interesting approach to identify attack
payloads through heuristics called Centrifuge
 Analysis of the use of special characters
 Ratio between the count of the word characters, spaces,
punctuation and the non word characters
 If <3.49 = malicious
 Normalization and stripping of any word character and
spaces including line breaks, tabs and carriage returns
 Regex check in default_filters.xml catches results
PHPIDS Lua Port Demo
Experimental Generic Detection Rules
 Two new experimental/beta generic detection rules
 Optional_rules/modsecurity_crs_40_experimental.conf
 Restricted Character Anomaly Usage
 Analyzes the number and type of meta-characters present in a
payload
 Testing thus far shows that the detection is good except for freeform text fields.
 Need to adjust the anomaly scoring for you own site
 Repetitive use of non-Word characters
 Currently alerts if there are 4 or more special characters found in a
row
Live CRS Demonstration Page
Live CRS Demonstration Page
http://www.modsecurity.org/demo/
CRS Demonstration Page
 Request will go through CRS page first and then we proxy
the request to the PHPIDS page
 http://demo.php-ids.org/
 We then inspect the inbound with the outbound and
provide results
 CRS detected an attack
 CRS did not find anything malicious but PHPIDS did
 Neither CRS nor PHPIDS found anything malicious
 A link is provided to report false negatives to our JIRA
ticketing system
 https://www.modsecurity.org/tracker/browse/CORERULES
 We have received >6700 attacks thus far
CRS Demonstration Page
Demonstration page Demo
Request Header Tagging
CRS Demonstration Page
 When in a distributed architecture, you can share WAF
data with downstream hosts
 Similar to SMTP SPAM data added to mime-headers
 optional_rules/modsecurity_crs_49_header_tagging.c
onf
 Maps to AppSensor Detection Point – RP2 (Suspicious
External User Behavior)
Example Header Tagging
GET /path/to/foo.php?test=1%27%20or%20%272%27=%272%27;-- HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.5)
Gecko/20091109 Ubuntu/9.10 (karmic) Firefox/3.5.5
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
X-WAF-Events: TX: / 999935-Detects common comment typesWEB_ATTACK/INJECTION-ARGS:test, TX:999923-Detects JavaScript
location/document property access and window access obfuscationWEB_ATTACK/INJECTION-REQUEST_URI_RAW, TX:950001WEB_ATTACK/SQL_INJECTION-ARGS:test
X-WAF-Score: Total=48; sqli=2; xss=
Connection: Keep-Alive
Future Directions
Future Directions
Preventing XSS with Content Injection
Javascript Sandboxing with Active Content Signatures
http://blog.modsecurity.org/2010/09/advanced-topic-of-theweek-xss-defense-via-content-injection.html
http://www.modsecurity.org/demo/demo-denynoescape.html
Implementing OWASP AppSensor Detection Points
We currently have some mappings for existing events
Will be using Lua to implement other
aggregate/behavioral/trend detection Points
Call for Community Help
 We have made great strides with CRS v2.0 but there is
still much work to be done
 Test out the CRS demo page and report any issues found
either to the mail-list or to JIRA
 Need Rule Documentation help
 Please sign up on our project mail-list if you want to help
 https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-corerule-set
Questions?
• Email – [email protected]
• Twitter - @ryancbarnett / @modsecurity