Using Undocumented CPU Behaviour to See into Kernel
Mode and Break KASLR in the Process
Anders Fogh and Daniel Gruss
August 4, 2016
1 / 64
About this presentation
This talk is about a class of microarchitectorial attacks
I
Not about software bugs
I
It is about CPU design as an attack vector
I
But not about Instruction Set Architecture
I
Focus on Intel x86-64 - applies to other architectures too
2 / 64
Take aways
Take aways
I
CPU design is security relevant
I
Prefetch instructions leak information
3 / 64
Take aways
Take aways
I
CPU design is security relevant
I
Prefetch instructions leak information
Exploit this to:
I
Locate a driver in kernel = defeat KASLR
I
Translate Virtual to physical addresses for other attacks
3 / 64
Introduction
Memory subsystem
Kernel Address-space Layout Randomization (KASLR)
Prefetch Side Channel
Prefetching the Kernel
Case study: Defeating Windows 7 KASLR
Case study: Exploiting direct-physical maps
Bonus material
4 / 64
Introduction
5 / 64
The chamber of secrets
NOTE
6 / 64
The chamber of secrets
NOTE
Using the PREFETCH instruction is recommended only if data does not fit in
cache.
6 / 64
The chamber of secrets
NOTE
Using the PREFETCH instruction is recommended only if data does not fit in
cache. Use of software prefetch should be limited to memory addresses that are
managed or owned within the application context.
6 / 64
The chamber of secrets
NOTE
Using the PREFETCH instruction is recommended only if data does not fit in
cache. Use of software prefetch should be limited to memory addresses that are
managed or owned within the application context. Prefetching to addresses that
are not mapped to physical pages can experience non-deterministic performance
penalty.
6 / 64
The chamber of secrets
NOTE
Using the PREFETCH instruction is recommended only if data does not fit in
cache. Use of software prefetch should be limited to memory addresses that are
managed or owned within the application context. Prefetching to addresses that
are not mapped to physical pages can experience non-deterministic performance
penalty. For example specifying a NULL pointer (0L) as address for a prefetch
can cause long delays.
6 / 64
The chamber of secrets (translation)
PLEASE
7 / 64
The chamber of secrets (translation)
PLEASE,
only use prefetch as Intel intends
7 / 64
The chamber of secrets (translation)
PLEASE,
only use prefetch as Intel intends,
or else Intel will be angry
7 / 64
The chamber of secrets (translation)
PLEASE,
only use prefetch as Intel intends,
or else Intel will be angry,
and there is no reason why anyone would measure the execution time.
7 / 64
The chamber of secrets (translation)
7 / 64
Whoami
I
Anders Fogh
I
Principal Security Researcher, GDATA Advanced Analytics
I
Playing with malware since 1992
I
Twitter: @anders_fogh
I
Email: [email protected]
8 / 64
Whoami
I
Daniel Gruss
I
PhD Student, Graz University of Technology
I
Currently intern at Microsoft Research Cambridge
I
Twitter: @lavados
I
Email: [email protected]
9 / 64
And the team
The research team
I
Clémentine Maurice
I
Moritz Lipp
I
Stefan Mangard
from Graz University of Technology
10 / 64
Memory subsystem
11 / 64
CPU Caches
Memory (DRAM) is slow compared to the CPU
I
buffer frequently used memory for the CPU
I
every memory reference goes through the cache
I
transparent to OS and programs
12 / 64
Memory Access Latency
cache hits
cache misses
Number of accesses
107
106
105
104
103
102
101
100
50
100
150
200
250
300
350
400
Access time in cycles
13 / 64
Data Caches
core 0
core 1
core 2
core 3
L1
L1
L1
L1
L2
L2
L2
L2
Last-level cache:
I
ring bus
shared memory shared is in
cache, across cores!
→ physically indexed
LLC
slice 0
LLC
slice 1
LLC
slice 2
I
need physical address to
manipulate
I
only one cache entry per
physical address
LLC
slice 3
14 / 64
Unprivileged cache maintainance
User programs can optimize cache usage:
I
prefetch: suggest CPU to load data into cache
I
clflush: throw out data from all caches
... based on virtual addresses
15 / 64
Caches: Software control
There are 5 prefetch instructions:
I
prefetcht0: suggest CPU to load data into L1
I
prefetcht1: suggest CPU to load data into L2
I
prefetcht2: suggest CPU to load data into L3
I
prefetchnta: suggest CPU to load data for non-temporal access
I
prefetchw: suggest CPU to load data with intention to write
actual behaviour varies between CPU models
16 / 64
Caches: Software control
The prefetch instructions are somewhat unusual
I
Hints – can be ignored by the CPU
I
Do not check privileges or cause exceptions
17 / 64
Virtual and physical addressing
Why address translation: Run multiple processes securely on a single CPU
I
Let applications run in their own virtual address space
I
Create exchangeable map from “virtual memory” to “physical memory”
I
Privileges are checked on memory accesses
I
Managed by the operating system kernel
18 / 64
Address translation on x86-64
CR3
PML4
PML4E 0
PML4E 1
··
·
#PML4I
··
·
PML4E 511
PDPT
PDPTE 0
PDPTE 1
··
·
#PDPTI
··
·
PDPTE 511
Page Directory
PDE 0
PDE 1
··
·
PDE #PDI
··
·
PDE 511
PML4I (9 b) PDPTI (9 b) PDI (9 b) PTI (9 b)
48-bit virtual address
Page Table
PTE 0
PTE 1
··
·
PTE #PTI
··
·
PTE 511
Offset (12 b)
4 KiB Page
Byte 0
Byte 1
··
·
Offset
··
·
Byte 4095
19 / 64
Address Translation Caches
Problem: translation tables are stored in physical memory
20 / 64
Solution: Address Translation Caches
Core 0
DTLB
PDE cache
PDPTE cache
Lookup direction
ITLB
Core 1
ITLB
PML4E cache
DTLB
PDE cache
PDPTE cache
PML4E cache
Page table structures in
system memory (DRAM)
21 / 64
Kernel Address-space Layout Randomization (KASLR)
22 / 64
Kernel is mapped in every process
Today’s operating systems:
Shared address space
User memory
Kernel memory
−1
0
context switch
23 / 64
Address-Space Layout Randomization (ASLR)
I
Kernel and drivers at randomized offsets in virtual memory
I
Mitigates code reuse attacks e.g. return-oriented-programming
I
Attacks based on read primitives or write primitives
24 / 64
Address-Space Layout Randomization (ASLR)
I
Kernel and drivers at randomized offsets in virtual memory
I
Mitigates code reuse attacks e.g. return-oriented-programming
I
Attacks based on read primitives or write primitives
I
But: leaking kernel/driver addresses defeats ASLR
24 / 64
Kernel direct-physical map
Physical memory
0
max. phys.
re
di
0
User
247
−247
Kernel
m
ct
ap
−1
Virtual address space
25 / 64
Kernel direct-physical map
Physical memory
0
max. phys.
re
di
0
User
247
−247
Kernel
m
ct
ap
−1
Virtual address space
Available on many operating systems / hypervisors
I OS X
I Linux
I BSD
I Xen PVM (Amazon EC2)
But not on Windows!
25 / 64
Prefetch Side Channel
26 / 64
Summary
1.
I
The kernel is mapped in every process
27 / 64
Summary
1.
I
The kernel is mapped in every process
2.
The prefetch instruction takes a virtual address as input
To manipulate L3 a physical address is needed
= The prefetch instruction must translate
I
I
27 / 64
Summary
1.
I
The kernel is mapped in every process
2.
The prefetch instruction takes a virtual address as input
To manipulate L3 a physical address is needed
= The prefetch instruction must translate
I
I
3.
The prefetch instruction does not check privileges
= Any address can be prefetched
I
27 / 64
Summary
1.
I
The kernel is mapped in every process
2.
The prefetch instruction takes a virtual address as input
To manipulate L3 a physical address is needed
= The prefetch instruction must translate
I
I
3.
The prefetch instruction does not check privileges
= Any address can be prefetched
I
4.
I
I
Translation is cached
Lookup searches caches in a fixed order
27 / 64
Summary
1.
I
The kernel is mapped in every process
2.
The prefetch instruction takes a virtual address as input
To manipulate L3 a physical address is needed
= The prefetch instruction must translate
I
I
3.
The prefetch instruction does not check privileges
= Any address can be prefetched
I
4.
I
I
Translation is cached
Lookup searches caches in a fixed order
Can we measure a time difference?
27 / 64
Execution time in cycles
There is a timing difference!
400
383
300
230
246
222
200
181
PDPT
PD
PT
Mapping level
cached P.
uncached P.
Idea: Would this also work on inaccessible kernel memory?
28 / 64
Prefetching the Kernel
29 / 64
A translation oracle
Definition of our translation oracle
I
Timing the prefetch instruction on an arbitrary address will recover the
translation level
30 / 64
Recovering a map of the kernel
Recovering /proc/pid/pagemap in 4 steps:
1. Recover mappings in PML4 by using the translation oracle on kernel
addresses
31 / 64
Recovering a map of the kernel
Recovering /proc/pid/pagemap in 4 steps:
1. Recover mappings in PML4 by using the translation oracle on kernel
addresses
2. Recover mappings in PDPT by using the translation oracle on kernel
addresses
31 / 64
Recovering a map of the kernel
Recovering /proc/pid/pagemap in 4 steps:
1. Recover mappings in PML4 by using the translation oracle on kernel
addresses
2. Recover mappings in PDPT by using the translation oracle on kernel
addresses
3. Recover mappings in PD by using the translation oracle on kernel addresses
31 / 64
Recovering a map of the kernel
Recovering /proc/pid/pagemap in 4 steps:
1. Recover mappings in PML4 by using the translation oracle on kernel
addresses
2. Recover mappings in PDPT by using the translation oracle on kernel
addresses
3. Recover mappings in PD by using the translation oracle on kernel addresses
4. Recover mappings in PT by using the translation oracle on kernel addresses
31 / 64
Recovering a map of the kernel
Recovering /proc/pid/pagemap in 4 steps:
1. Recover mappings in PML4 by using the translation oracle on kernel
addresses
2. Recover mappings in PDPT by using the translation oracle on kernel
addresses
3. Recover mappings in PD by using the translation oracle on kernel addresses
4. Recover mappings in PT by using the translation oracle on kernel addresses
Complete process takes from seconds to hours depending on how pages are
actually mapped by the operating system.
31 / 64
The address-translation oracle
The address-translation oracle tell us whether virtual address p and p̄ map to the
same physical address
1. Use clflush to remove p from cache
32 / 64
The address-translation oracle
The address-translation oracle tell us whether virtual address p and p̄ map to the
same physical address
1. Use clflush to remove p from cache
2. Use prefetch to load p̄ into cache
32 / 64
The address-translation oracle
The address-translation oracle tell us whether virtual address p and p̄ map to the
same physical address
1. Use clflush to remove p from cache
2. Use prefetch to load p̄ into cache
3. time access of p. If fast it was cached: p: maps to the same physical memory
as p̄
Beware! prefetch is a hint!
32 / 64
Timing instructions
The CPU may reorder instructions – a look at rdtscp
instruction 1
rdtscp
instruction 2
rdtscp
instruction 3
33 / 64
Timing instructions
The CPU may reorder instructions – a look at rdtscp
instruction 1
rdtscp
3
instruction 2
3
rdtscp
3
instruction 3
33 / 64
Timing instructions
The CPU may reorder instructions – a look at rdtscp
instruction 1
rdtscp
7
instruction 2
rdtscp
7
instruction 3
33 / 64
Timing instructions
The CPU may reorder instructions – a look at mfence
instruction 1
mfence
instruction 2
mfence
instruction 3
34 / 64
Timing instructions
The CPU may reorder instructions – a look at mfence
instruction 1
mfence
3
instruction 2
3
mfence
3
instruction 3
34 / 64
Timing instructions
The CPU may reorder instructions – a look at mfence
instruction 1
7
mfence
instruction 2
7
mfence
instruction 3
34 / 64
Timing instructions
The CPU may reorder instructions
instruction 1
cpuid
instruction 2
cpuid
instruction 3
but not over cpuid!
35 / 64
Timing the prefetch instruction
The CPU may reorder prefetch instruction – a look at rdtscp
prefetch
rdtscp
prefetch
rdtscp
prefetch
36 / 64
Timing the prefetch instruction
The CPU may reorder prefetch instruction – a look at rdtscp
prefetch
rdtscp
3
prefetch
3
rdtscp
3
prefetch
36 / 64
Timing the prefetch instruction
The CPU may reorder prefetch instruction – a look at rdtscp
prefetch
rdtscp
3
prefetch
rdtscp
3
prefetch
36 / 64
Timing the prefetch instruction
The CPU may reorder instructions – a look at mfence
prefetch
mfence
prefetch
mfence
prefetch
37 / 64
Timing the prefetch instruction
The CPU may reorder instructions – a look at mfence
prefetch
mfence
3
prefetch
3
mfence
3
prefetch
37 / 64
Timing the prefetch instruction
The CPU may reorder instructions – a look at mfence
prefetch
mfence
3
prefetch
mfence
3
prefetch
37 / 64
Timing instructions
Combine instructions in clever procedures to
I prevent reordering of prefetch if necessary
I
avoid noise from cpuid if possible
38 / 64
Timing instructions
Combine instructions in clever procedures to
I prevent reordering of prefetch if necessary
= mfence rdtscp cpuid target instruction cpuid rdtscp mfence
I
avoid noise from cpuid if possible
38 / 64
Timing instructions
Combine instructions in clever procedures to
I prevent reordering of prefetch if necessary
= mfence rdtscp cpuid target instruction cpuid rdtscp mfence
I
avoid noise from cpuid if possible
= mfence cpuid rdtscp target instruction rdtscp cpuid mfence
38 / 64
Timing instructions
Combine instructions in clever procedures to
I prevent reordering of prefetch if necessary
= mfence rdtscp cpuid target instruction cpuid rdtscp mfence
I
avoid noise from cpuid if possible
= mfence cpuid rdtscp target instruction rdtscp cpuid mfence
We use either the prefetchnta or prefetcht2 instructions and a mov instruction
for memory access
38 / 64
Case study: Defeating Windows 7 KASLR
39 / 64
Windows 7 Memory layout
I
HAL and kernel located bewetween
I
I
I
start: 0xffff f800 0000 0000
end : 0xffff f87f ffff ffff
Kernel drivers
I
I
start: 0xffff f880 0000 0000
end : 0xffff f8ff ffff ffff
40 / 64
Windows 7 Breaking KASLR
1. Map the drivers address space using translation recovery attack
41 / 64
Windows 7 Breaking KASLR
1. Map the drivers address space using translation recovery attack
2. Evict the page translation caches: Sleep() and/or access large memory
buffer
41 / 64
Windows 7 Breaking KASLR
1. Map the drivers address space using translation recovery attack
2. Evict the page translation caches: Sleep() and/or access large memory
buffer
3. Perform a syscall to targeted driver
41 / 64
Windows 7 Breaking KASLR
1. Map the drivers address space using translation recovery attack
2. Evict the page translation caches: Sleep() and/or access large memory
buffer
3. Perform a syscall to targeted driver
4. Time prefetch(PageAddress)
41 / 64
Windows 7 Breaking KASLR
1. Map the drivers address space using translation recovery attack
2. Evict the page translation caches: Sleep() and/or access large memory
buffer
3. Perform a syscall to targeted driver
4. Time prefetch(PageAddress)
5. Repeat 2,3,4 for all pages found in 1
Fastest average access time is right address.
41 / 64
Avg. execution time in cycles
Locate Kernel Driver (defeat KASLR)
120
110
100
90
0
4,000
8,000
12,000
Page offset in kernel driver region
42 / 64
Case study: Exploiting direct-physical maps
43 / 64
Kernel exploits
I
Overwrite return address
→ jump to userspace code
I
Overwrite stack pointer
→ switch to userspace stack
44 / 64
Mitigating kernel exploits
I
Jump to userspace code? Nope! Hardware prevents that.
= Supervisor-mode execution prevention (SMEP)
I
Switch to userspace stack? Nope! Hardware prevents that.
= Supervisor-mode access prevention (SMAP)
45 / 64
Kernel direct-physical map
Physical memory
0
max. phys.
ct
ap
m
re
di
0
User
247
−247
Kernel
−1
Virtual address space
46 / 64
Evading the mitigation
I
Get direct-physical-map address of userspace address
→ jump/switch there
Known as “ret2dir” attacks Kemerlis et al. 2014
47 / 64
Mitigating the evasion
I
Getting rid of direct-physical map?
48 / 64
Mitigating the evasion
I
Getting rid of direct-physical map? Apparently not.
→ Do not leak physical addresses to user
48 / 64
Circumventing the mitigation
Prefetching via direct-physical map
I
use known address or translation recovery attack to find the
direct-physical-map
I
find user mode address in with direct-physical map using
address-translation oracle
49 / 64
Circumventing the mitigation
Min. access latency in cycles
Prefetching via direct-physical map
250
200
150
100
0
20
40
60 80 100 120 140 160 180 200 220 240
Page offset in direct-physical map
50 / 64
Prefetching via direct-physical map
I
immediately leaks a direct-physical map address
→ no information leak necessary (compared to ret2dir)
I
if direct-physical map offset is known
→ leaks physical address
51 / 64
Prefetching via direct-physical map
I
works on Linux
I
works on OSX
I
works on Xen PVM (on Amazon EC2)
does not work on Windows
I
I
no direct-physical map ;)
52 / 64
Cache side-channel attacks
Powerful side channel in the cloud
I
infer user input
I
crypto key recovery
I
cross-VM, cross-core, even cross-CPU
I
any architecture
53 / 64
Cache mapping
35
physical address
17
6
tag
set
0
offset
30
H
11
I
slice function H unknown
I
reverse-engineered by Hund et al.
2
2013; Maurice et al. 2015; Inci et al.
2015; Yarom et al. 2015
→ we need the physical address
line
slice 0
slice 1
slice 2
slice 3
54 / 64
Rowhammer
Rowhammer: yet another attack requiring physical address information
I
Rowhammer: bit flip at a random location in DRAM
I
exploitable → gain root privileges Seaborn and Dullien 2015
55 / 64
Rowhammer
“It’s like breaking into an apartment by repeatedly slamming a neighbor’s door until the vibrations
open the door you were after” – Motherboard Vice
DRAM bank
11111111111111
11111111111111
11111111111111
11111111111111
...
11111111111111
row buffer
56 / 64
Rowhammer
“It’s like breaking into an apartment by repeatedly slamming a neighbor’s door until the vibrations
open the door you were after” – Motherboard Vice
DRAM bank
11111111111111
activate
11111111111111
11111111111111
11111111111111
...
copy
11111111111111
row buffer
56 / 64
Rowhammer
“It’s like breaking into an apartment by repeatedly slamming a neighbor’s door until the vibrations
open the door you were after” – Motherboard Vice
DRAM bank
11111111111111
11111111111111
11111111111111
activate
11111111111111
...
11111111111111
copy
row buffer
56 / 64
Rowhammer
“It’s like breaking into an apartment by repeatedly slamming a neighbor’s door until the vibrations
open the door you were after” – Motherboard Vice
DRAM bank
11111111111111
activate
11111111111111
11111111111111
11111111111111
...
copy
11111111111111
row buffer
56 / 64
Rowhammer
“It’s like breaking into an apartment by repeatedly slamming a neighbor’s door until the vibrations
open the door you were after” – Motherboard Vice
DRAM bank
11111111111111
11111111111111
11111111111111
activate
11111111111111
...
11111111111111
copy
row buffer
56 / 64
Rowhammer
“It’s like breaking into an apartment by repeatedly slamming a neighbor’s door until the vibrations
open the door you were after” – Motherboard Vice
DRAM bank
11111111111111
bit flips in row 2!
11111111111111
10111110101111
11111111111111
...
11111111111111
row buffer
56 / 64
How is DRAM organized?
57 / 64
How is DRAM organized?
channel 0
channel 1
57 / 64
How is DRAM organized?
back of DIMM: rank 1
channel 0
front of DIMM:
rank 0
channel 1
57 / 64
How is DRAM organized?
back of DIMM: rank 1
channel 0
front of DIMM:
rank 0
chip
channel 1
57 / 64
DRAM organization example
bank 0
chip
row 0
row 1
row 2
...
row 32767
row buffer
I
bits in cells in rows
I
access: activate row,
copy to row buffer
I
cells leak → refresh
necessary
I
cells leak faster upon
proximate accesses
58 / 64
DRAM mapping functions on a DDR4 system
BG0
BG1
Rank
BA0
BA1
... 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 ...
Ch.
Again: based on physical addresses
59 / 64
Summary
I
I
Defeat SMAP/SMEP through direct-physical map
Leak physical addresses
I
I
Perform cache attacks
Perform Rowhammer attacks
60 / 64
Black Hat Sound Bytes.
I
CPU Design is security relevant (prefetch leaks significant information)
I
We can locate a driver in the kernel and thus break KASLR
I
We can break SMAP/SMEP and get physical addresses to assist other attacks
61 / 64
Using Undocumented CPU Behaviour to See into Kernel
Mode and Break KASLR in the Process
Anders Fogh and Daniel Gruss
August 4, 2016
62 / 64
Bonus material
63 / 64
Stronger Kernel Isolation
Today’s operating systems:
Shared address space
User memory
Kernel memory
−1
0
context switch
Stronger kernel isolation:
User address space
User memory
context switch
Not mapped
0
Not mapped
switch
addr. space
0
−1
Interrupt
dispatcher
Kernel memory
Kernel address space
−1
64 / 64
Bibliography I
Hund, Ralf et al. (2013). “Practical Timing Side Channel Attacks against Kernel Space ASLR”. In:
S&P’13.
Inci, Mehmet Sinan et al. (2015). “Seriously, get off my cloud! Cross-VM RSA Key Recovery in a
Public Cloud”. In: Cryptology ePrint Archive, Report 2015/898, pp. 1–15.
Kemerlis, Vasileios P et al. (2014). “ret2dir: Rethinking kernel isolation”. In: USENIX Security
Symposium, pp. 957–972.
Maurice, Clémentine et al. (2015). “Reverse Engineering Intel Complex Addressing Using
Performance Counters”. In: RAID.
Seaborn, Mark and Thomas Dullien (2015). “Exploiting the DRAM rowhammer bug to gain kernel
privileges”. In: Black Hat 2015 Briefings.
Yarom, Yuval et al. (2015). “Mapping the Intel Last-Level Cache”. In: Cryptology ePrint Archive,
Report 2015/905, pp. 1–12.
65 / 64