MQTT and Message Metadata
Version 1.0
Working Draft 03
16 November 2015
Technical Committee:
OASIS Message Queuing Telemetry Transport (MQTT) TC
Chairs:
Raphael J Cohn ([email protected]), Individual
Richard J Coppen ([email protected]), IBM
Editor:
Andrew Banks ([email protected]), IBM
Ian Craggs ([email protected]), IBM
Related work:
This document is related to:
 MQTT Version 3.1.1. Edited by Andrew Banks and Rahul Gupta. Latest
version: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqttv3.1.1.html.
Abstract:
This document describes message formats used in MQTT.
Status:
This Working Draft (WD) has been produced by one or more TC Members;
it has not yet been voted on by the TC or approved as a Committee Note
Draft. The OASIS document Approval Process begins officially with a TC
vote to approve a WD as a Committee Note Draft. A TC may approve a
Working Draft, revise it, and re-approve it any number of times as a
Committee Note Draft.
URI patterns:
Initial publication URI:
http://docs.oasis-open.org/mqtt/msgfmts/v1.0/cnd01/msgfmts-v1.0cnd01.doc.
Permanent “Latest version” URI:
http://docs.oasis-open.org/mqtt/msgfmts/v1.0/msgfmts-v1.0.doc.
(Managed by OASIS TC Administration; please don’t modify.)
This is intended as a Non-Standards Track Work Product.
The patent provisions of the OASIS IPR Policy do not apply.
Copyright © OASIS Open 2015. All Rights Reserved.
All capitalized terms in the following text have the meanings assigned to them in the OASIS
Intellectual Property Rights Policy (the "OASIS IPR Policy"). The full Policy may be found at the
OASIS website.
This document and translations of it may be copied and furnished to others, and derivative
works that comment on or otherwise explain it or assist in its implementation may be prepared,
copied, published, and distributed, in whole or in part, without restriction of any kind, provided
that the above copyright notice and this section are included on all such copies and derivative
works. However, this document itself may not be modified in any way, including by removing
the copyright notice or references to OASIS, except as needed for the purpose of developing any
document or deliverable produced by an OASIS Technical Committee (in which case the rules
applicable to copyrights, as set forth in the OASIS IPR Policy, must be followed) or as required to
translate it into languages other than English.
This document and the information contained herein is provided on an "AS IS" basis and OASIS
DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY
WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY OWNERSHIP
RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR
PURPOSE.
msgmetadata-v1.0-wd03
Non-Standards Track
Working Draft 03
Copyright © OASIS Open 2015. All Rights Reserved.
16 November 2015
Page 2 of 9
[Type the document title]
The limited permissions granted above are perpetual and will not be revoked by OASIS or its
successors or assigns.
This is intended as a Non-Standards Track Work Product.
The patent provisions of the OASIS IPR Policy do not apply.
Table of Contents
1
Introduction ............................................................................................................................. 3
1.1 References (non-normative).................................................................................................. 3
1.2 Section Level 2 ....................................................................................................................... 4
1.2.1 Section Level 3 ................................................................................................................ 4
1.3 Section Level 2 ....................................................................................................................... 7
2
Heading .................................................................................................................................... 8
Appendix A.
Acknowledgments ................................................................................................... 8
Appendix B.
Some Appendix ........................................................................................................ 9
B.1.1 Sub-subsidiary Appendix Section.................................................................................... 9
Appendix C.
Revision History ....................................................................................................... 9
1
2
3
4
1 Introduction
MQTT conveys compact but limited data about its messages. This committee note discusses the
need to extend the metadata associated with messages.
5
1.1 References (non-normative)
6
[Jira-249]
7
8
Add expiry capabilities to MQTT. 09 February 2015. OASIS MQTT TC Jira-249. Jira-249 Add expiry
capabilities to MQTT.
9
[Jira-256]
10
11
Message Format indication. 11 February 2015. OASIS MQTT TC Jira-256. Jira-256 Message
Format indication.
12
[Jira-273]
13
14
Comments on request / reply TC note. 06 August 2015. OASIS MQTT TC Jira-273. Jira-273
Comments on request / reply TC note.
msgmetadata-v1.0-wd03
Non-Standards Track
Working Draft 03
Copyright © OASIS Open 2015. All Rights Reserved.
16 November 2015
Page 3 of 9
[Type the document title]
B.1 Subsidiary Appendix Section ................................................................................................. 9
15
[Jira-269]
16
17
MQTT-SN Feature: Topic Registration. 02 July 2015. OASIS MQTT TC Jira-269. jira-269 MQTT-SN
Feature: Topic Registration.
18
[Jira-277]
19
20
Comments on Message Metadata WD2. 21 September 2015. OASIS MQTT TC Jira-277. Jira-277
Comments on Message Metadata WD2.
21
1.2 Requirements
22
The data that is carried by MQTT as part of a message is currently limited to:
23
The message payload
24
The topic name of the message
25
The Qos of the message
26
Whether the message should be, or was retained by the server
27
Whether the message is a duplicate
28
29
Some messages carry a message identifier, but this is not usually useful to a producing
or consuming application
30
31
Examine some examples of metadata not already carried by MQTT and categorise the attributes
of the metadata.
32
1.2.1 Message payload type
33
34
35
36
See [Jira-256]. MQTT messages are bytes. It has been proposed that some description of how to
interpret the bytes should be added to the protocol. For example this might indicate whether
the bytes represent a string or a number. The recipient of the message may choose to interpret
the bytes in a different ways by knowing what they represent.
37
38
There are a large number of possible message types and it would be burdensome for an
application to have to support all possible types.
39
40
41
42
43

The metadata should place no open ended capability on either the producer or
consumer of a message. It should be possible to write a producer or consumer client
application which fully supports every possible value of the metadata. So, for the
payload type it should be possible to write a generic consumer which can handle every
possible value of payload type.
44
45
46
47

The producer and consumer applications may operate successfully by only using a
subset of the metadata and by only allowing a subset of the possible values of the
metadata. For example the producer and consumer applications may expect to only
ever send String data.
msgmetadata-v1.0-wd03
Non-Standards Track
Working Draft 03
Copyright © OASIS Open 2015. All Rights Reserved.
16 November 2015
Page 4 of 9
[Type the document title]
This is intended as a Non-Standards Track Work Product.
The patent provisions of the OASIS IPR Policy do not apply.
48
49
50

Client and server implementations do not validate values of metadata that they do not
interpret. Consequently the consuming application may choose to validate that the data
it receives is the payload type claimed.
51
52
53
54

The producer and consumer applications may operate successfully without examining
the metadata. The producer and consumer could, for example, continue to send a String
as a set of bytes and not set or payload type to String, this would mean an intermediary
might not understand the payload.
55
56
57
58
59

The client or server implementation may not need to understand the metadata it is
passing between the producing and consuming clients. In this case the server must pass
the metadata unchanged between the producing and consuming clients. Consequently a
server cannot simply receive payload type String messages and emit them as payload
type bytes.
60
61
62

A minimal server implementation may safely ignore all of the metadata, other than
passing it on to the consumer. This enables minimal server implementations to be
created at a cost of having partial implementations.
63

The data flowing across the network should be compact.
64
65
1.2.2 Time to live
66
67
68
69
70
71
See [Jira-249]. Messages may be useful for a limited amount of time, for instance because they
convey the current temperature. System resources can be saved by discarding messages which
are no longer useful, as indicated by the time to live. If the server stores a message before
delivering it to a consumer it might reduce the time to live of the outbound message by the
amount of time that it is stored in the server. The receiver should never observe a value of the
time to live which is negative.
72
73
74
75

Message metadata which is interpreted by the client or server implementation must be
well defined such that it is possible to write a client or server implementation which
fully supports all types of metadata. So, the time to live must be defined such that it
always represents a practical amount of time, like a positive number of milliseconds.
76
77
78

Client or server implementations may safely ignore metadata if they wish to provide a
reduced implementation. A server might forward the time to live value to the consumer
unchanged and not actually expire the messages.
79
80

Client and server implementations should validate the values of metadata that they
interpret. For example, messages with a negative time to live should be rejected.
81
82
83
84

Server implementations should pass all metadata to the eventual consumers, however
the metadata passed on may not be exactly that which they received. If the producer
set a time to live, then the consumer would always see a time to live value in the
messages it received.
msgmetadata-v1.0-wd03
Non-Standards Track
Working Draft 03
Copyright © OASIS Open 2015. All Rights Reserved.
16 November 2015
Page 5 of 9
[Type the document title]
This is intended as a Non-Standards Track Work Product.
The patent provisions of the OASIS IPR Policy do not apply.
This is intended as a Non-Standards Track Work Product.
The patent provisions of the OASIS IPR Policy do not apply.
85
1.2.3 Correlator
86
87
See [Jira-273]. In a request response messaging pattern the request message needs to carry a
correlator so that the responder can indicate which request it is responding to.
88
89

The metadata may be relevant to only one message at a time. It must be possible to
determine which message the metadata relates to.
90
91
92

The client or server implementations should not need to store unbounded amounts of
metadata. The conditions under which the metadata can be safely discarded should be
well defined.
93
1.2.4 Summary of requirements
94
See [Jira-277].
1. The meaning of all of the metadata fields should be specified and closed. Neither the
client and server implementations can add their own meaning. The client
implementation in particular should prevent direct access by the client application to
the metadata because this would allow the client application to subvert its meaning.
99
100
101
2. The producer and consumer applications (as opposed to the client and server
implementations) may use their own conventions to communicate metadata, but this is
outside the scope of the MQTT specification.
102
103
104
3. The producer or consumer applications are not required to use any of the metadata
values. The default values and default behavior written in the specification will allow
interoperation at the level of function provided by MQTT Version 3.1.1.
105
106
4. The client and server implementations are not obliged to validate the metadata,
although they can reject packets if they find the metadata is invalid.
107
108
109
110
5. The specification will describe how the server may modify metadata fields, however the
minimal implementation of passing on every field unchanged to each consumer will
always be allowed. A server may not remove metadata fields which it does not
implement.
111
112
113
114
115
6. A minimal client implementation might use the default value and behavior for all
metadata. A minimal server implementation might simply pass on all metadata
unchanged. This implies that an intermediary may not be able to predict what a server
implementation might do with a message unless it knows whether the server
implements the metadata behavior.
116
7. The networks formats are compact.
117
118
8. This specification of the metadata does not permit values that cannot be implemented
by the client or server, hence an intermediary can always interpret the metadata values.
119
120
9. It is possible for the client implementation, server implementation or an intermediary to
identify exactly what metadata is associated with a packet.
msgmetadata-v1.0-wd03
Non-Standards Track
Working Draft 03
Copyright © OASIS Open 2015. All Rights Reserved.
16 November 2015
Page 6 of 9
[Type the document title]
95
96
97
98
This is intended as a Non-Standards Track Work Product.
The patent provisions of the OASIS IPR Policy do not apply.
121
122
123
10. It is possible to determine when the client and server implementations no longer need
to store metadata. The implementations will not require additional configuration to
perform memory management.
124
125
11. The primary use of metadata is related to publication of messages, however, carrying of
metadata related to other functions may ultimately be needed.
1.3 Possible solutions.
127
128
1. Augment the MQTT packets to carry metadata either as variable header fields, encoded
into a single variable header field or as part of the payload.
129
130
2. Maintain a cache of metadata templates in both the client and server and add an
indication into the packet as to which template the metadata should be taken from.
131
1.4 Proposed specification
132
133
134
135
136
A new protocol version will indicate the presence of metadata. The scheme proposed is to
augment the PUBLISH packet and maintain a cache of metadata templates for as long as the
client server connection lasts. A template identifier indicates which cached template for the
metadata is to be modified ( if necessary ) and then used to apply the metadata to the
publication.
137
1.4.1 Placement of metadata within MQTT PUBLISH packets
138
139
The metadata will take to place of the Topic Name in the PUBLISH packet, the topic name will be
carried as one of the metadata fields encoded as described in 1.4.2 below.
140
141
142
If extra fields are needed in the CONNECT packet these will be provided by using the existing
scheme of positional fields in the payload with their presence or absence indicated by new flag
bits in the variable header
143
144
For other packets, the same encoding as for the PUBLISH packet could be used for a field in the
variable header.
145
1.4.2 Encoding of metadata within the MQTT PUBLISH packet
146
The publish packet changes from:
147
03 xx <TopicName> <PacketId> <Payload>
148
To
149
03 xx <MetaData> <PacketId> <Payload>
150
151
The metadata contains the topic name and is formed as follows:
152
TemplateId(Byte) , Id/[Value] pairs, delimiter(Byte).
msgmetadata-v1.0-wd03
Non-Standards Track
Working Draft 03
Copyright © OASIS Open 2015. All Rights Reserved.
16 November 2015
Page 7 of 9
[Type the document title]
126
153
154
155
156
A cache of up to 256 templates are retained by the client and server only the changed Id/Value
fields are flowed and replaced in the cache in the sender and receiver before processing the
message. The cache is cleared when the network connection closes. The delimiter indicates that
no more metadata is present.
157
158
If Id is flowed as a Multi Byte length, like the current remaining length field, then the first 128 id’s
occupy a single byte but we have an extendable set of possible id’s.
159
As an example, the starter set of field Id Value definitions might be.
Id
Value
Initial Value
0
Delimiter
None
1
Topic Name (UTF-8)
None, must be flowed on first use of each
template.
2
Message Payload Type (Byte) 0= bytes,
1=UTF-8 String, 2=Integer, four bytes.
0=bytes.
3
Time to live in seconds, four bytes.
FFFFFFFF (forever).
4
Correlator, four bytes.
Zero.
160
161
162
The shortest metadata value is Id,0 two bytes and repeats the metadata set by the previous use
of the Id. This creates a capability similar to that described in [Jira-269], topic registration.
163
164
A minimal implementation might send 0,1,TopicName,0 in place of the current topic name in
every publish packet. This extends the length of the publish packet by 3 additional bytes.
165
166
A more advanced implementation would omit the Topic Name for a repeated publication resulting
a net saving of network bytes if the topic name is greater than 1 character.
167
168
169
170
171
2 Heading
Text.
3 Acknowledgments
The following individuals have participated in the creation of this specification and are gratefully
acknowledged:
172
Participants:
173
174
[Participant Name, Affiliation | Individual Member]
[Participant Name, Affiliation | Individual Member]
175
msgmetadata-v1.0-wd03
Non-Standards Track
Working Draft 03
Copyright © OASIS Open 2015. All Rights Reserved.
16 November 2015
Page 8 of 9
[Type the document title]
This is intended as a Non-Standards Track Work Product.
The patent provisions of the OASIS IPR Policy do not apply.
This is intended as a Non-Standards Track Work Product.
The patent provisions of the OASIS IPR Policy do not apply.
4 Some Appendix
Text.
178
4.1 Subsidiary Appendix Section
179
Text.
180
4.1.1 Sub-subsidiary Appendix Section
181
text.
182
5
Revision History
Revision
Date
Editor
Changes Made
1
06 July 2015
Andrew Banks
Add requirements
2
27 July 2015
Andrew Banks
Include Requirement for lightweight /
optional implementation. Enumerate
requirements.
3
16
November
2015
Andrew Banks
Update requirements, add proposed
specification.
183
msgmetadata-v1.0-wd03
Non-Standards Track
Working Draft 03
Copyright © OASIS Open 2015. All Rights Reserved.
16 November 2015
Page 9 of 9
[Type the document title]
176
177