2 Request/Reply Message Exchange Patterns and MQTT Version 1.0 3 Working Draft 04 4 02 November 2015 5 6 Technical Committee: OASIS Message Queuing Telemetry Transport (MQTT) TC 7 8 9 Chairs: Raphael J Cohn ([email protected]), Individual Richard J Coppen ([email protected]), IBM 1 10 11 Editor: Shawn McAllister ([email protected]), Solace Systems 12 13 14 15 16 Related work: This document is related to: MQTT Version 3.1.1. Edited by Andrew Banks and Rahul Gupta. Latest version: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqttv3.1.1.html. 17 18 19 20 21 22 23 24 Abstract: This document is a Working Draft document for the MQTT Technical committee. It describes context, motivation, requirements, and proposals for implementing a request/reply message exchange pattern for applications using the MQTT 3.1.1 protocol. It is intended to focus discussion within the MQTT Working Group to allow the group to decide on whether and how to address these requirements—with changes to the MQTT 3.1.1 specification, a Committee Note or some other mechanism. 25 26 27 28 29 30 31 Status: This Working Draft (WD) has been produced by one or more TC Members; it has not yet been voted on by the TC or approved as a Committee Note Draft. The OASIS document Approval Process begins officially with a TC vote to approve a WD as a Committee Note Draft. A TC may approve a Working Draft, revise it, and re-approve it any number of times as a Committee Note Draft. 32 33 34 35 URI patterns: Initial publication URI: http://docs.oasis-open.org/mqtt/reqreply/v1.0/cnd01/reqreply-v1.0cnd01.doc. This is intended as a Non-Standards Track Work Product. The patent provisions of the OASIS IPR Policy do not apply. 36 37 Permanent “Latest version” URI: http://docs.oasis-open.org/mqtt/reqreply/v1.0/reqreply-v1.0.doc. 38 (Managed by OASIS TC Administration; please don’t modify.) 39 41 Copyright © OASIS Open 2015. All Rights Reserved. 42 43 44 All capitalized terms in the following text have the meanings assigned to them in the OASIS Intellectual Property Rights Policy (the "OASIS IPR Policy"). The full Policy may be found at the OASIS website. 45 46 47 48 49 50 51 52 53 This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published, and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this section are included on all such copies and derivative works. However, this document itself may not be modified in any way, including by removing the copyright notice or references to OASIS, except as needed for the purpose of developing any document or deliverable produced by an OASIS Technical Committee (in which case the rules applicable to copyrights, as set forth in the OASIS IPR Policy, must be followed) or as required to translate it into languages other than English. 54 55 The limited permissions granted above are perpetual and will not be revoked by OASIS or its successors or assigns. 56 57 58 59 60 This document and the information contained herein is provided on an "AS IS" basis and OASIS DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY OWNERSHIP RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. reqreply-v1.0-wd02 Non-Standards Track Working Draft 02 Copyright © OASIS Open 2015. All Rights Reserved. 12 August 2015 Page 2 of 15 [Type the document title] 40 This is intended as a Non-Standards Track Work Product. The patent provisions of the OASIS IPR Policy do not apply. 61 Table of Contents 62 1 64 1.1 References (non-normative).................................................................................................. 4 2 Problem Definition................................................................................................................... 5 65 2.1 Motivation ............................................................................................................................. 5 66 2.2 Requirements ........................................................................................................................ 6 67 2.2.1 Requirements on Interactions ........................................................................................ 6 68 2.2.2 Requirements for Topology Support .............................................................................. 7 69 3 Candidate Solution #1 .............................................................................................................. 9 70 3.1 Overview ................................................................................................................................ 9 71 3.2 Protocol Considerations ...................................................................................................... 11 72 4 Candidate Solution #2 ............................................................................................................ 12 73 Acknowledgments ................................................................................................. 13 74 Summary of Previous Discussions ......................................................................... 14 75 Use HTTP for Request/Response ......................................................................................... 14 76 Request/Reply is not the Same as Synchronous ................................................................. 14 77 Request/Reply is not the Same as Remote Procedure Call (RPC) ....................................... 14 78 Revision History ..................................................................................................... 15 79 80 81 reqreply-v1.0-wd02 Non-Standards Track Working Draft 02 Copyright © OASIS Open 2015. All Rights Reserved. 12 August 2015 Page 3 of 15 [Type the document title] 63 Introduction ............................................................................................................................. 4 This is intended as a Non-Standards Track Work Product. The patent provisions of the OASIS IPR Policy do not apply. 1 Introduction 83 84 85 This document is a Working Draft document for the MQTT Technical committee. It contains all matter related to adding a request/reply capability for applications using the MQTT 3.1.1 protocol. 86 87 88 89 This document contains the motivation, requirements and solution options for providing such a capability. It is intended to focus discussion within the MQTT Working Group to allow the group to decide on an appropriate way to address these requirements – whether with changes to the MQTT 3.1.1 specification or via a Committee Note or some other mechanism. 90 1.1 References (non-normative) 91 [MQTT] 92 93 MQTT Version 3.1.1. Edited by Andrew Banks and Rahul Gupta. Latest version: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html. [Type the document title] 82 reqreply-v1.0-wd02 Non-Standards Track Working Draft 02 Copyright © OASIS Open 2015. All Rights Reserved. 12 August 2015 Page 4 of 15 94 2 Problem Definition 95 96 97 This section describes motivation and draft requirements related to defining the request/reply problem addressed by this document. In Appendix B we describe issues raised in early discussions relating to request/reply 98 99 The TC plans to clarify the problem statement and then address potential solutions in later Working Drafts of this Note. 100 2.1 Motivation 101 102 103 104 105 106 The MQTT protocol [MQTT] supports a publish/subscribe message exchange pattern where a published message is delivered to zero or more consumers based on subscription but where the publisher does NOT receive any message in response to its published message. Publish/subscribe works very well for applications where an actor publishes “notifications” of an event, status change, or sensor reading, and where many consumers can subscribe to and receive that information. 107 108 109 However, there are many common situations where a request/reply message exchange pattern is required and this is not explicitly supported by [MQTT] nor has guidance been provided on how such a message exchange pattern can be performed by applications or APIs using [MQTT]. 110 Three examples of where such a mechanism is needed are: 111 112 113 114 When an IoT device needs to “read” or download some information from a service running in a datacenter. In this case, a device would like to send a message on a topic and receive one (or perhaps more) replies to this particular message. This is similar to an HTTP GET operation. 115 116 117 118 When an application in a datacenter needs to “read” some information immediately from a device without waiting some timed period for the new value to be sent from the device. This is like an HTTP GET but in this case it is sent from the datacenter application to the IoT device – the reverse direction of a typical HTTP GET. 119 120 121 122 When an application in a datacenter needs to “set” some value onto an IoT device and needs confirmation that this set was successful. Somewhat like an HTTP POST but in this case it is sent from the datacenter application to the IoT device – the reverse direction of a typical HTTP POST. 123 124 125 126 127 128 While it is possible for MQTT 3.1.1 clients to perform request/reply exchanges using privatelyagreed conventions between the requester and the responder, it is desirable to elevate this from many private conventions to a single public convention for the following reasons: 1. A common mechanism for performing request/reply allows for interoperability between any conforming applications without pair-wise application conventions. reqreply-v1.0-wd02 Non-Standards Track Working Draft 02 Copyright © OASIS Open 2015. All Rights Reserved. 12 August 2015 Page 5 of 15 [Type the document title] This is intended as a Non-Standards Track Work Product. The patent provisions of the OASIS IPR Policy do not apply. This is intended as a Non-Standards Track Work Product. The patent provisions of the OASIS IPR Policy do not apply. 2. It fosters application simplicity and robustness. Each set of applications need not design and implement their own convention “reinventing the wheel” and dealing with common challenges but rather can simply re-use one public mechanism. 3. It allows for request/reply interoperability with non-MQTT clients in an applicationindependent manner which can be implemented by common infrastructure. 135 2.2 Requirements 136 137 The specific requirements of the request/reply mechanism are discretely defined here for discussion and agreement before proceeding to analyze solution options. 138 2.2.1 Requirements on Interactions 139 140 141 142 143 We state the requirements in the context of a “requester” and a “responder” as shown in Figure 1 below. Any MQTT client can be a requester or a responder for any given message. The “requester” is the MQTT client that initiates the request by sending a request message. The “responder” is the MQTT client that receives the request message and sends a response message. 144 145 This section specifies the requirements for when the requester and responder are both MQTT clients. The case where one client is a non-MQTT client is addressed in Section 2.2.2. 146 147 F IGURE 1 MQTT REQUESTER AND RESPONDER 148 EDITOR’S NOTE: NEED BETTER CONTEXT AND REASONS WHY THESE ARE REQUIREMENTS 149 150 151 152 153 154 155 156 1. Support symmetric Request/Reply Interactions. Support request/reply interactions where any MQTT client can be a either a requester or a responder for any particular message. 157 158 159 2. Support Request/Reply with routing to many applications Support request/reply where the request may be routed to many clients (pub/sub) and where the requester may receive many replies to its one request. . 160 161 162 3. Support asynchronous Request/Reply The request/reply mechanisms must be fully asynchronous. This is as opposed to HTTP which imposes a synchronous interaction. Many requests may be outstanding at any one [Narrative: This supports device autonomously requesting information from a datacenter application and allows a datacenter application to request information from a device or effect some change on a device or cause some action by the device as and when the datacenter application decides to do so, without requiring polling (and coupling).] reqreply-v1.0-wd02 Non-Standards Track Working Draft 02 Copyright © OASIS Open 2015. All Rights Reserved. 12 August 2015 Page 6 of 15 [Type the document title] 129 130 131 132 133 134 This is intended as a Non-Standards Track Work Product. The patent provisions of the OASIS IPR Policy do not apply. 163 164 165 166 167 time over a single MQTT connection. Applications and APIs can implement synchronous and/or blocking behaviors on top of this asynchronous protocol if they desire. 168 169 170 171 172 4. Allow the requestor to correlate replies with the original request Since many requests may be outstanding at the same time on an MQTT connection, the defined mechanisms must allow the requestor to correlate a reply message with the original request (instance) that was sent (or the request state) and must allow for replies to be received in a different order than the order in which the requests were sent. 173 174 175 5. Allow the server to scale well To ensure scalability of many clients and many requests in concurrently progress, it is highly desirable that the request/reply interaction be stateless in the MQTT server. 176 177 178 6. Allow any QoS at the Client’s discretion Both the request portion and the reply portion of the message exchange pattern must be able to use any of QoS 0, 1 or 2 at the client’s discretion 179 180 7. 181 2.2.2 Requirements for Topology Support 182 183 184 The following topologies must be supported for Request/Reply; see Figure 1 above. 185 186 187 188 189 190 Requestor and Responder are both MQTT clients that are connected to the same or different MQTT servers. Use of different servers could be when MQTT bridges are used to support MQTT servers on gateways, trains, buses, etc. 191 192 193 194 195 196 Topology 2: [Type the document title] Allow the QoS of the reply to differ from the QoS of the request The QoS of the request need not match the QoS of the reply. Topology 1: EDITOR’S NOTE: DO WE NEED DIFFERENT FIGURES FOR CLARITY? I’d also prefer something like “the Device and Application are MQTT clients…” Once client is an MQTT client but the second client uses some other non-MQTT protocol and in fact may or may not be connected to the same server as the first client. This topology is depicted in Figure 2. reqreply-v1.0-wd02 Non-Standards Track Working Draft 02 Copyright © OASIS Open 2015. All Rights Reserved. 12 August 2015 Page 7 of 15 This is intended as a Non-Standards Track Work Product. The patent provisions of the OASIS IPR Policy do not apply. 197 198 8. Well-known, application-independent syntax and semantics of request/reply information In this topology, it must be possible for the “MQTT-to-Non-MQTT” interworking function (IWF) to be NOT application-specific. That is, there must be a standard method for the protocol interworking function to find in the MQTT packet the information required to interwork the MQTT request/reply with the request/reply information of the Non-MQTT protocol. Note that there is no requirement to actually specify the protocol interactions between MQTT and any other protocol. Motivation: Motivation for this requirement on Topology 2 is to avoid the need for application+messagingspecific bridges to support request/reply in this topology. Message brokers with complex internal structure that implement multiple protocols could then provide request/reply capabilities between MQTT clients and clients of another protocol using the broker in a completely standard manner – without application-specific gateways. This is an important topology because while it is expected that MQTT will be widely used by IoT devices, it is not expected that it be as widely used by datacenter applications and thus this messaging protocol interworking must be offered as simply as possible “out of the box” by products. reqreply-v1.0-wd02 Non-Standards Track Working Draft 02 Copyright © OASIS Open 2015. All Rights Reserved. 12 August 2015 Page 8 of 15 [Type the document title] 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 F IGURE 2 MQTT AND NON -MQTT REQUESTER /RESPONDER T OPOLOGY This is intended as a Non-Standards Track Work Product. The patent provisions of the OASIS IPR Policy do not apply. 221 3 Candidate Solution #1 222 This section outlines a mechanism that addresses the requirements of Section 2.2. 223 There are three components to the solution as follows: 1. Reply-to address in request messages. 2. Correlation-ID in the request and reply messages. 3. Uniqueness of reply-to address. 227 228 229 230 At this stage, rather than propose specific text for the mechanism, we describe how these components are used in a request/reply interaction and provide some protocol considerations. This text is offered to convey the main concepts of the solution, not as finalized text suitable for an OASIS public document. 231 3.1 Overview 232 233 234 235 236 237 238 The request and reply messages are normal PUBLISH packets sent by the Requester and Responder respectively. What makes them different than normal “one-way publish” messages that MQTT 3.1.1 supports today is the inclusion of additional fields in the packets as described below. That said, the MQTT server can be completely oblivious to the (end-to-end) request/reply message exchange pattern if it chooses to be, which is a desirable attribute in terms of backward compatibility, impact on existing server implementations, performance and scalability of the solution. 239 Reply-To address 240 241 242 243 244 245 In order for a MQTT PUBLISH packet to indicate that it is a request and for the Responder to know where to send the reply, the Requester adds a Reply-To field to a PUBLISH packet. The Reply-To address is any valid MQTT Topic Name. The Reply-To address is carried in the PUBLISH packet (containing the request message) to the (zero or more) Responders unchanged by any MQTT server. The requester will have subscribed in advance using a Topic Filter that matches the Topic Name in the Reply-To field in order to receive reply messages. 246 247 248 249 The Responder receives the request message and performs the functions required based on the contained Application Message. The Responder returns a reply message by sending a PUBLISH packet with an Application Message in the payload and the Topic Name of the PUBLISH packet set to the Reply-To address from the request message. 250 251 Since the Requester will have subscribed appropriately in advance, the reply message is routed back to the Requester. 252 The request message and the reply message can be of any QoS. 253 Using a Topic Name as the addressing method has several advantages: 254 255 - This “state” information is carried with the message and thus does not require additional mapping or otherwise that increase complexity or reduce the scalability of reqreply-v1.0-wd02 Non-Standards Track Working Draft 02 Copyright © OASIS Open 2015. All Rights Reserved. 12 August 2015 Page 9 of 15 [Type the document title] 224 225 226 This is intended as a Non-Standards Track Work Product. The patent provisions of the OASIS IPR Policy do not apply. - the solution for either the MQTT server or any client. Once a request message is delivered to the Responder, the state is passed along with it and is deleted from the MQTT server. From a security point of view, any existing access controls that a broker may implement that restrict publish or subscribe capabilities based on topic apply here as well. 261 Correlation-ID 262 263 264 The Requestor may choose to include in its request PUBLISH packet an additional field called the Correlation-ID. The Correlation-ID is meaningful only to the publisher and is used to allow the publisher to match a received reply message with the state from the original request message. 265 266 267 268 269 The Correlation-ID is sent in the request message and passed transparently by the MQTT server to the Responder(s) as part of the PUBLISH packet. The Responder returns the Correlation-ID from the request message in the associated reply message which is passed transparently by the MQTT server to the Requester. The Requester may then use this Correlation-ID to match the received reply message with state from the associated request message. 270 Uniqueness of Reply-To Address 271 272 273 274 275 While the Requester can provide any valid Topic Name in the Reply-To field of a request message, in most cases it is desirable that this Topic Name be guaranteed to be unique among all clients of the MQTT server and in fact unique among all clients of the set of MQTT servers in a particular domain. Leaving this determination to client configuration, while possible, is very error-prone, complex and difficult to secure. 276 277 278 279 280 We propose providing a mechanism for an MQTT client to acquire a unique Topic Name from its MQTT server that the client can use in its Reply-To field to receive replies. The MQTT server ensures that this Topic Name is unique among all clients within a given domain. We could simply specify the uniqueness requirements that a MQTT server must implement, or require a particular structure for uniformity – for example, that Topic Name be structured as 281 <Server Root>/<client-unique-part> 282 283 although clearly multiple options exist. We advocate prescribing a particular structure for uniformity. 284 285 286 In terms of the mechanism to acquire this address, if we had a request/reply capability within MQTT the client could just send a request to a well-known topic and receive its unique Topic Name via the reply. 287 288 289 290 291 292 One method to simulate this request/reply with mechanisms currently in MQTT 3.1.1 is for the client to subscribe to a special, reserved Topic Name and receive its unique Reply-To Topic Name in the body of a message published to it from the MQTT server like a Retained message would be. The body of the message is different for each MQTT client, but the mechanism is identical for all and follows the existing pattern of subscribing to a Topic Filter where a single matching Retained message exists. reqreply-v1.0-wd02 Non-Standards Track Working Draft 02 Copyright © OASIS Open 2015. All Rights Reserved. 12 August 2015 Page 10 of 15 [Type the document title] 256 257 258 259 260 293 3.2 Protocol Considerations 294 Request / Reply Messages 295 296 The structure for encoding the Reply-To field and the Correlation-ID field are simple and should follow existing precedents in MQTT 3.1.1. 297 298 299 300 301 302 It makes most sense, from a protocol consistency point of view, to encode these optional fields in the Variable Header portion of the PUBLISH message before the Payload (containing the Application Message). While this can easily be done, the key is signaling the existence of these (new) fields in a PUBLISH packet so they can be processed by those entities who can or need to process them, but can be safely ignored by clients that do not support the request/reply capability – and do so in a manner that is compatible with the current MQTT 3.1.1 specification. 303 304 Carrying any new metadata (this is not request/reply specific) in a PUBLISH packet is challenging given the current wording of the MQTT 3.1.1 specification since: 305 306 307 308 309 - There are no un-used fields to work with in the fixed portion of the PUBLISH packet The Remaining Length field contains the length of the combined Variable Header and the Payload. This is unfortunate given that the Variable Header has a fixed encoding depending on the QoS of the PUBLISH packet, so adding new fields to the Variable Payload as specified cannot be done in a backward compatible manner. 310 311 312 Given these constraints, we do not have a proposal that would allow any new metadata to be added to a PUBLISH message in a backward compatible manner. Hopefully the original authors of this capability have a proposal on how to accomplish this. 313 314 315 316 317 318 319 Failing this, the next option of least disruption is to add a new MQTT Control Packet Type to support PUBLISH packets that contain metadata. This is a bit more involved as we would need to use the only reasonable value left (15) to indicate an extension where we would define an extended Control Packet Type, the first of which is PUBLISH2 which could contain metadata, but that could support other types in the future. The metadata support mechanisms of PUBLISH2 would need to be generic to allow addition of new metadata over time in a backward compatible manner. 320 Uniqueness of Reply-To Address 321 322 323 Discovery of a client’s (unique) Reply-To address following a method described in Section 3.1 requires no changes to the MQTT3.1.1 protocol itself – only assignment of the reserved Topic Filter to subscribe to and a description of the client and server procedures involved. 324 reqreply-v1.0-wd02 Non-Standards Track Working Draft 02 Copyright © OASIS Open 2015. All Rights Reserved. 12 August 2015 Page 11 of 15 [Type the document title] This is intended as a Non-Standards Track Work Product. The patent provisions of the OASIS IPR Policy do not apply. This is intended as a Non-Standards Track Work Product. The patent provisions of the OASIS IPR Policy do not apply. 4 Candidate Solution #2 326 EDITOR’S NOTE 327 328 This section will describe at a high level a second candidate solution to the problem. Analysis must address the requirements and compare to other solutions with pros/cons. [Type the document title] 325 reqreply-v1.0-wd02 Non-Standards Track Working Draft 02 Copyright © OASIS Open 2015. All Rights Reserved. 12 August 2015 Page 12 of 15 This is intended as a Non-Standards Track Work Product. The patent provisions of the OASIS IPR Policy do not apply. Acknowledgments 329 330 331 The following individuals have participated in the creation of this specification and are gratefully acknowledged: 332 Participants: 333 David Horton, Solace Systems 334 Ken Borgendale, IBM 335 Shawn McAllister, Solace Systems 336 William Cox, Individual Member [Type the document title] 337 reqreply-v1.0-wd02 Non-Standards Track Working Draft 02 Copyright © OASIS Open 2015. All Rights Reserved. 12 August 2015 Page 13 of 15 This is intended as a Non-Standards Track Work Product. The patent provisions of the OASIS IPR Policy do not apply. Summary of Previous Discussions 338 341 This section summarizes previous discussions related to using request/reply message exchange patterns in an MQTT environment Use HTTP for Request/Response 342 343 344 345 346 It was proposed that devices use HTTP to provide a request/reply capability, and that devices use MQTT when they need pub/sub. The main reason that this option is rejected is because of requirement #1 where a datacenter application needs to send a request to a device. This would require that the device support an HTTP server stack and that its IP address be known to and reachable from the datacenter application. 347 348 349 350 Another softer reason is that this would indeed require devices to support a dual stack and that the datacenter infrastructure would be significantly more complex with both MQTT and HTTP servers, duplicate authentication mechanisms, etc. Having a single mechanism for publish/subscribe as well as request/reply overall is simpler. 351 Request/Reply is not the Same as Synchronous 352 353 354 355 There was discussion of the complexity and lack of scalability of request/reply mechanisms but this may have been due to a misunderstanding that the request/reply mechanism would be synchronous. In fact as per requirement #3, it must be possible for both synchronous and asynchronous modes to be offered by an MQTT API. 356 357 358 From a protocol point of view, the request and reply messages are “asynchronous” in that after a client sends a request message, any other MQTT packets can be sent on the connection by the client before the reply is received. 359 Request/Reply is not the Same as Remote Procedure Call (RPC) 360 361 362 We view the request/reply mechanism defined in this document as being a superset of the RPC capability and that RPC could be built on top of this request/reply. In particular, some differences are: 363 364 RPC is synchronous, as viewed by the application – request/reply can be synchronous or asynchronous 365 366 RPC receives at most one reply per request whereas the request/reply mechanism could receive many replies for a single request. 367 368 As such, RPC will be dealt with separately from the request/reply capability defined in this document but is expected to use the capabilities resulting from this document. reqreply-v1.0-wd02 Non-Standards Track Working Draft 02 Copyright © OASIS Open 2015. All Rights Reserved. 12 August 2015 Page 14 of 15 [Type the document title] 339 340 This is intended as a Non-Standards Track Work Product. The patent provisions of the OASIS IPR Policy do not apply. Revision History 369 Revision Date Editor Changes Made 01 15 June 2015 Shawn McAllister Initial version for discussion. Captured: motivation, proposed requirements. 02 11 August 2015 William Cox Restructured including moving rejected proposals into an appendix. Clarified text, made requirements more visible with following discussion. 03 22 Sept 2015 Shawn McAllister Draft updates to address MQTT-273 and MQTT-274 04 2 Nov 2015 Shawn McAllister Added solution initial proposal text in section 3.1 [Type the document title] 370 reqreply-v1.0-wd02 Non-Standards Track Working Draft 02 Copyright © OASIS Open 2015. All Rights Reserved. 12 August 2015 Page 15 of 15
© Copyright 2025 Paperzz