Computer Science
A Fault Model and Mutation Testing
of Access Control Policies
Evan Martin and Tao Xie
Department of Computer Science
North Carolina State University
Outline
• Motivation
• XACML
• Policy Testing Techniques
– Coverage Criteria
– Request Generation
– Request Selection
• Fault Model & Mutation Testing
• Experimental Results
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
2
Motivation
• Digital information is
– Easy to access
– Easy to search
• Sensitive information requires access control
mechanisms
• A growing trend is to specify access control
policies in a specification language such as
XACML
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
3
Problem
• How to ensure the correct specification of
access control policies?
– What you specify is what you get, but not
necessarily what you want
• Systematic testing of access control policies
– Complements policy verification, which requires
properties and may not support full policy features
– Just like software testing + software verification
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
4
Software Testing
Policy Testing
Software Testing
Test
Inputs
Requests
Program
Policy
Computer Science
Test
Outputs
Expected
Outputs
Responses
Expected
Responses
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
5
XACML Policy Structure
• eXtensible Access Control Markup Language
– OASIS standard XML syntax for specifying policies, requests, and
responses
– A flexible and expressive language but complex and verbose
Key concepts
• A Policy Set holds other policies or policy sets.
• A Policy is expressed as a set of rules.
• A Rule have targets and a set of conditions that determine if
the rule applies to a given request.
• Both rule and policy Combining Algorithms exist to reconcile
conflicts.
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
6
XACML Example
<?xml version="1.0" encoding="UTF-8"?>
<PolicySet xmlns="urn:oasis:names:tc:xacml:1.0:policy" PolicySetId="college"
Policy Set
PolicyCombiningAlgId="urn:oasis:names:tc:xacml:1.0:policy-combining-algorithm:permit-overrides">
<Description>A College Policy on Grades</Description>
<Target>
<Subjects> <AnySubject /> </Subjects>
Target
<Resources> <AnyResource /> </Resources>
<Actions> <AnyAction /> </Actions>
Policy
</Target>
<Policy PolicyId="fac"
RuleCombiningAlgId="urn:oasis:names:tc:xacml:1.0:rule-combining-algorithm:permit-overrides">
<Description>Faculty Policy</Description>
<Target>
<Subjects>
<Subject>
<SubjectMatch
MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<AttributeValue
DataType="http://www.w3.org/2001/XMLSchema#string">
Faculty
Target
</AttributeValue>
<SubjectAttributeDesignator AttributeId="role"
DataType="http://www.w3.org/2001/XMLSchema#string" />
</SubjectMatch>
</Subject>
</Subjects>
<Resources>
<AnyResource />
</Resources>
<Actions>
<AnyAction />
</Actions>
</Target>
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
7
XACML Example
<Rule RuleId="fac-assign-view-grades" Effect="Permit">
<Target>
Rule
<Subjects> <AnySubject /> </Subjects>
<Resources>
<Resource>
<ResourceMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">ExternalGrades</AttributeValue>
<ResourceAttributeDesignator AttributeId="resource-class"
DataType="http://www.w3.org/2001/XMLSchema#string" />
</ResourceMatch>
</Resource>
<Resource>
<ResourceMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">InternalGrades</AttributeValue>
<ResourceAttributeDesignator AttributeId="resource-class"
DataType="http://www.w3.org/2001/XMLSchema#string" />
</ResourceMatch>
</Resource>
Target
</Resources>
<Actions>
<Action>
<ActionMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">Assign</AttributeValue>
<ActionAttributeDesignator AttributeId="command"
DataType="http://www.w3.org/2001/XMLSchema#string" />
</ActionMatch>
</Action>
<Action>
<ActionMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">Receive</AttributeValue>
<ActionAttributeDesignator AttributeId="command"
DataType="http://www.w3.org/2001/XMLSchema#string" />
</ActionMatch>
</Action>
</Actions>
</Target>
</Rule>
</Policy>
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
8
XACML Example
<Policy PolicyId="stu" RuleCombiningAlgId="urn:oasis:names:tc:xacml:1.0:rule-combining-algorithm:permit-override
<Description>Student Policy</Description>
<Target>
<Subjects>
<Subject>
<SubjectMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">Student</AttributeValue>
<SubjectAttributeDesignator AttributeId="role" DataType="http://www.w3.org/2001/XMLSchema#string" />
</SubjectMatch>
</Subject>
</Subjects>
<Resources><AnyResource /></Resources>
<Actions><AnyAction /></Actions>
</Target>
<Rule RuleId="stu-recieve-extgrades" Effect="Permit">
<Target>
<Subjects><AnySubject /></Subjects>
<Resources>
<Resource>
<ResourceMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">ExternalGrades</AttributeValue>
<ResourceAttributeDesignator AttributeId="resource-class"
DataType="http://www.w3.org/2001/XMLSchema#string" />
</ResourceMatch>
</Resource>
</Resources>
<Actions>
<Action>
<ActionMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">Receive</AttributeValue>
<ActionAttributeDesignator AttributeId="command"
DataType="http://www.w3.org/2001/XMLSchema#string" />
</ActionMatch>
</Action>
</Actions>
</Target>
</Rule>
</Policy>
</PolicySet>
Rules can have a more complicated condition tag here
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
9
Coverage Definition
[ICICS 06]
• Rationale: like in program testing, when the policy
part containing a fault is not “covered”, the error is
often not exposed.
• Given a request q and a policy P, we say q covers a
rule m in P if m contributes to the decision of q.
• The rule m contributes to the decision of q if all
Cond are satisfied by q.
– The sequence of rules and combining algorithms may also
affect coverage.
• Rule coverage of P by requests Q =
#rules covered by at least one request in Q
#rules in P
• Similarly define policy coverage/condition coverage
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
10
Random Request Generation
[ICICS 06]
• The example policy:
– Subjects: Student, Faculty
– Actions: Assign, Receive
– Objects: External Grades, Internal Grades
• Model the set of attribute values as a vector of
bits and randomize the bits
Student Faculty Assign Receive ExtGrades IntGrades
1
0
1
Computer Science
0
May 12, 2007
0
1
WWW 2007, Banff,
Alberta, Canada
11
Random Request Generation
[ICICS 06]
• The example policy:
– Subjects: Student, Faculty
– Actions: Assign, Receive
– Objects: External Grades, Internal Grades
• Model the set of attribute values as a vector of
bits and randomize the bits
Student Faculty Assign Receive ExtGrades IntGrades
0
1
1
Computer Science
0
May 12, 2007
1
1
WWW 2007, Banff,
Alberta, Canada
12
Request Generation via Change-Impact
[SESS 07]
Analysis (Cirg)
Access
Control
Policy
1. Version
Synthesis
Requests
2. Change-impact
analysis
3. Request
generation
Counter
examples
Synthesize versions
(all-to-negate-one)
for Margrave
[Kisler et al. 05]
Computer Science
Policy
Versions
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
13
Greedy Algorithm for
Request Selection
[ICICS 06]
Policy
Requests
Increased
Coverage?
Yes
Reduced
Request
Set
No
Discard
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
14
Fault Model and Mutation Testing
• Fault model used to model things that could go
wrong when constructing a policy
• Fault model is the underlying foundation of
mutation testing [DeMillo et al. 78] (similar to fault
injection):
– Policy is iteratively mutated to produce numerous
mutants – each containing one fault
• Usages
– Measure fault-detection effectiveness of test
generation or test selection
– Select tests
WWW 2007, Banff,
Computer Science
May 12, 2007
Alberta, Canada
15
Policy Mutation Testing
Policy
Requests
Mutation
Operators
Computer Science
Responses
Mutator
Differ?
Mutant
Policy
Mutant
Responses
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
Mutant
Killed!
16
Mutation Operators
• Each operator mutates a
different policy element:
policy set, policy, rule,
condition, and/or their
associated targets and
effects.
Computer Science
Op
Description
PSTT
Policy Set Target True
PSTF
Policy Set Target False
PTT
Policy Target True
PTF
Policy Target False
RTT
Rule Target True
RTF
Rule Target False
RCT
Rule Condition True
RCF
Rule Condition False
CPC
Change Policy Combining algorithm
CRC
Change Rule Combining algorithm
CRE
Change Rule Effect
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
17
Change Rule Effect (CRE) Example
IF (faculty AND assign AND grades)
Permit
ELSE IF (student AND receive AND grades)
Permit
ELSE
Deny
• The CRE mutation operator is performed on each rule
and changes the decision effect (Permit  Deny)
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
18
Change Rule Effect (CRE) Example
IF (faculty AND assign AND grades)
Deny
ELSE IF (student AND receive AND grades)
Permit
ELSE
Deny
• The CRE mutation operator is performed on each rule
and changes the decision effect (Permit  Deny)
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
19
Experiment
• How strong is the correlation between the basic
coverage criteria and fault-detection capability?
• Does test selection based on the coverage
criteria produce reduced request sets with low
loss of fault-detection capability?
• What are the individual characteristics of each
mutation operator?
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
20
Metrics
•
•
•
•
•
Policy, rule, and condition coverage
Test count, t
Reduced test count, tred
Mutant-killing ratio, m
Reduced mutant-killing ratio, mred
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
21
Policies used in the experiment
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
22
Basic Coverage Results
• Cirg performs at least as well as the random set for rule coverage except for
the mod-fedora policy because of a policy error.
• Average # of selected random tests is smaller than Cirg
• Random achieves 0% coverage on the conference policy
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
23
Mutant-killing Results
• Cirg outperforms the random technique in terms of fault-detection capability
with a far fewer number of requests
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
24
Mutant-killing ratios by subjects
0% coverage => 0% mutant-kill
Coverage criteria
not bad for selection
not great
either
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
25
Mutant-killing ratios by operators
Likely equivalent
mutants
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
26
Conclusions
• Policy testing complements policy verification
in assuring policy correctness.
– Just like software testing + software verification
• We have developed coverage criteria, test
selection, and test generation.
• This paper presents fault model and mutation
testing for policies
– Used to measure fault-detection effectiveness of test
generation or test selection
– Used to select tests
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
27
Computer Science
Questions?
May 12, 2007
WWW 2007, Banff, Alberta, Canada
28
Hypothesis
• We can achieve a significant reduction in
request-set size for large randomly generated
request sets while maintaining equivalent
policy, rule, and condition coverage.
• Reducing a request set based on coverage will
not proportionately decrease its fault detection
capability.
• Request generation via Change-impact analysis
(Cirg) will have a higher fault-detection
capability.
Computer Science
May 12, 2007
WWW 2007, Banff,
Alberta, Canada
29
XACML Coverage Criteria
[ICICS 06]
• Policy coverage – A policy is covered by a
request if the policy is applicable to the request
and the policy is encountered before the PDP
has fully resolved the decision for the request.
• Rule coverage – A rule is covered by a request if
the rule is applicable to the request and the rule
is encountered before the PDP has fully
resolved the decision for the request.
• Condition coverage – A condition must be
encountered and evaluate to true and false to be
fully covered.
WWW 2007, Banff,
Computer Science
May 12, 2007
Alberta, Canada
30