One-Shot STags
John Carrier
Adaptec
August 04, 2004
John Carrier, Adaptec
1
Agenda
• Problem Statement
• Proposals
• Recommendations
August 04, 2004
John Carrier, Adaptec
2
Problem Statement
• STags may be valid for more than one RDMA access, which may
leave the ULP open to attack from the remote peer.
• The ULP may be especially vulnerable from attacks through RDMA
Writes
– The remote peer indicates completion of the RDMA through an RDMA
Send type message.
– When the local ULP receives this completion message, it may assume
that it again has ownership of its sink buffer.
– However, unless the STag itself is invalidated, the remote peer may
overwrite the buffer even as the local ULP is processing its contents.
• This remote access may be inadvertent or deliberate.
• IPsec cannot address this problem because the error is generated
by the connection peer, not a man in the middle.
August 04, 2004
John Carrier, Adaptec
3
RDMA Write with Explicit ULP Invalidation
Data Sink
Data Source
ULP
untagged receive
buffers
RNIC
PostRQ
(buffers)
Q0(add buffers)
Register
Sink Buffer
Q0(add buffers)
ULP
PostRQ
(buffers)
untagged receive
buffers
TPT(add)
STag=X
sink buffer
RNIC
PostSQ
(Request, Sink=X
RDMA Send Msg (S
ink=X)
)
Receive
Request
TPT(add)
, TO=N)
RDMA Write (Sink=X
lete)
mp
RDMA Send Msg (co
Receive
Complete
Invalidate
STag=X
August 04, 2004
Register
Source Buffer
STag=Y
PostSQ
=N)
(Snk=X,Src=Y,TO
source buffer
PostSQ
(Complete)
TPT(invalidate)
John Carrier, Adaptec
4
Peer Attack of Data Sink Buffer
Data Sink
Data Source
ULP
RNIC
PostRQ
(buffers)
untagged receive
buffers
Q0(add buffers)
Register
Sink Buffer
PostSQ
(Request, Sink=X
Q0(add buffers)
untagged receive
buffers
=X)
mplete)
RDMA Send Msg (co
TPT(add)
Receive
Complete
, TO=N)
RDMA Write (Sink=X
Receive
Request
Register
Source Buffer
STag=Y
PostSQ
=N)
(Snk=X,Src=Y,TO
PostSQ
(Complete)
, TO=N)
RDMA Write (Sink=X
August 04, 2004
PostRQ
(buffers)
RDMA Send Msg (S
ink
)
TPT(add)
Invalidate
STag=X
ULP
TPT(add)
STag=X
sink buffer
RNIC
source buffer
Register
ur
So ce Buffer B
STag=Z
PostSQ
=N)
(Snk=X,Src=Z,TO
source buffer B
TPT(invalidate)
John Carrier, Adaptec
5
Proposed Solutions
• Caitlin Bestler proposed ‘one-shot’ STags as a means of
limiting the scope of STags to a single RDMA
transaction.
• Current architectures, however, enable persistent STags
that may be used for multiple RDMA events.
– Some protocols such as SCSI, will rely on persistent STags to
enable targets to DMA data to initiator buffers in the way targets
find most efficient (ie as it becomes available)
• Reflector discussions focused on mechanisms where the
RNIC could invalidate the STag before the ULP
processed the completion
– Send & Invalidate
– Receive & Invalidate
– RDMA Write & Invalidate
August 04, 2004
John Carrier, Adaptec
6
Send & Invalidate
• DDP already includes a Send & Invalidate message that
allows the remote peer to include an STag in a Send
Message which the local RNIC can invalidate before the
local ULP processes the completion.
• The original intent of this message was to reduce a
roundtrip to the RNIC during completion processing. If
the RNIC invalidated the STag, then the local peer would
not have to write to the RNIC after receiving the
completion.
• The mechanism, however, effectively closes remote
access to the STag and removes the vulnerability to
further attack from the remote peer.
August 04, 2004
John Carrier, Adaptec
7
Implicit Invalidation: Send & Invalidate
Data Sink
Data Source
ULP
untagged receive
buffers
RNIC
PostRQ
(buffers)
Q0(add buffers)
Register
Sink Buffer
PostSQ
(Request, Sink=X
Q0(add buffers)
ULP
PostRQ
(buffers)
untagged receive
buffers
TPT(add)
STag=X
sink buffer
RNIC
RDMA Send Msg (S
ink
=X)
)
TPT(add)
, TO=N)
RDMA Write (Sink=X
date Msg
RDMA Send & Invali
(complete, STag=X)
Receive
Request
Register
Source Buffer
STag=Y
PostSQ
=N)
(Snk=X,Src=Y,TO
source buffer
PostSQ
)
(Complete,STag=X
TPT(invalidate)
Receive
Complete
August 04, 2004
John Carrier, Adaptec
8
Receive & Invalidate
• Caitlin suggested that the local peer should be in control
of invalidating its own STags.
– Send & Invalidate leaves that control with the remote peer.
– Unless the ULP explicitly describes the use of Send & Invalidate
(as iSER does), it is possible that the message will not be used
correctly and sink buffers will be vulnerable.
• For ULPs that know which untagged receive buffer will
be used to complete an RDMA Write, Caitlin proposed a
new mechanism where the ULP included an STag when
it posted the receive buffer.
• When the RNIC consumes the untagged buffer, it
retrieves the STag associated with it, and then
invalidates the STag for the Data Sink buffer.
August 04, 2004
John Carrier, Adaptec
9
Implicit Invalidation: Receive & Invalidate
Data Sink
ULP
RNIC
Register
Sink Buffer
sink buffer
untagged
response buffer
Data Source
RNIC
TPT(add)
Q0(buffers)
ULP
PostRQ
(buffers)
untagged buffers
STag=X
PostRQ
(buffer, STag=X)
Q0(buffer+STag)
Post Request
(Sink=X)
RDMA Send Msg (S
ink=X)
Request
TPT(add)
, TO=N)
RDMA Write (Sink=X
mp
RDMA Send Msg (co
lete)
Register
Source Buffer
STag=Y
PostSQ
=N)
(Snk=X,Src=Y,TO
PostSQ
(complete)
source buffer
Q0(retrieve bufptr+STag)
TPT(invalidate)
Receive
Complete
August 04, 2004
John Carrier, Adaptec
10
RDMA Write & Invalidate
• In both Send & Invalidate and Receive & Invalidate the
RNIC invalidates the STag following an RDMA Send
Message.
• It is possible to have the RNIC invalidate the STag
following the RDMA Write Message.
– The RNIC “knows” that a Write Message is complete
– After ACKing the last byte of the message, it could invalidate the
STag associated with the RDMA write.
• The problem with this proposal is that the remote peer
may require multiple RDMA Writes to fill the Data Sink
buffer.
– iSER, for example, could not use this mechanism
August 04, 2004
John Carrier, Adaptec
11
Multiple Use of a Data Sink Stag
Data Sink
Data Source
ULP
untagged receive
buffers
RNIC
PostRQ
(buffers)
Q0(add buffers)
Register
Sink Buffer
PostSQ
(Request, Sink=X
Q0(add buffers)
ULP
PostRQ
(buffers)
untagged receive
buffers
TPT(add)
STag=X
sink buffer
RNIC
RDMA Send Msg (S
ink
=X)
)
TPT(add)
=N)
, TO
RDMA Write (Sink=X
TPT(add)
Receive
Request
Register
Source Buffer
STag=Y
PostSQ
=N)
(Snk=X,Src=Y,TO
Register
Source BufferB
source buffer
STag=Z
source buffer B
, TO
RDMA Write (Sink=X
TPT(invalidate)
=M)
ate Msg
RDMA Send & Invalid
(complete, STag=X)
PostSQ
=M)
(Snk=X,Src=Z,TO
PostSQ
)
(complete, STag=X
Receive
Complete
August 04, 2004
John Carrier, Adaptec
12
Recommendations
• Send & Invalidate is the only general purpose solution
that can be validated on the wire
• Receive & Invalidate may be useful to specific classes of
applications.
– Two well known protocols (SDP, iSER), for example, do not have
such tight control of receive resources as to be able to associate
STags for an RDMA transaction with a specific untagged buffer
to hold its completion.
– Furthermore, enabling this mechanism requires a non-trivial
change to the RNIC Interface in order for the RNIC to store a
STags in a queue of STag buffer pointers.
• Write & Invalidate restricts use of RDMA Write
August 04, 2004
John Carrier, Adaptec
13
Discussion
• Are one-shot STags really necessary?
• Would it suffice to describe the pitfalls of not using
Send&Invalidate?
August 04, 2004
John Carrier, Adaptec
14