OData Delta Query Protocol Design
Draft 1.3
2013/1/23
Overview
This document describes extensions to the OData protocol to support incremental maintenance of local
(i.e., cached) results through "Delta Queries".
Requirements
-no per-client state required in service
-idempotent on client
-follows odata interaction, flow, RESTful principles
Scenarios
The Delta Query Protocol supports synchronizing data from a single service to multiple clients. Coupled
with the ability to submit updates from the client to the service, this can be used to provide a solution
for keeping client's local data in sync with a single service.
The Delta Query protocol is not designed for multi-master synchronization scenarios.
Design
The Delta Query Protocol defines a pull-model protocol for clients to request subsequent changes to a
response retrieved from the service.
Changes include newly created, updated, or deleted resources and links as well as resources that have
been added to, or removed from, the result due to changes to properties on which filters have been
applied.
Clients can poll the service for changes periodically or as required, or capable services can provide
change notifications to alert the client when changes are available to be pulled. The definition of such
change notifications is outside of the scope of this version of the protocol.
Delta Query Support
An OData service may support clients requesting changes to zero, all, or a subset of entitysets within the
service.
For those entitysets for which a service supports result maintenance, it may support maintenance of
results for arbitrary queries against the entityset or it may require that any filters be applied solely to
immutable (i.e., key) fields.
Delta Links
For any request for which the OData Service can enumerate changes the service SHOULD support result
maintenance, if requested, by including a "delta link" url in place of the next link on the last page of
results returned.
A response MAY contain a next page link, a delta link, or neither, but MUST NOT contain both.
Clients can use this delta link to query the service for changes that have occurred to the result since the
initial request.
Delta links MUST only be returned upon requests whose responses contain an entity or a collection of
zero or more entities. Requests for individual property values MUST NOT include delta links. For
example, ~/Customers('ALFKI')/ContactName.
Requesting Delta Links
Services MAY return delta links in response to any request that supports tracking changes. An OData
client MAY request that a service return delta links by including a Preference value of odata-requestdeltas in the Prefer Header of the request. Services that support deltas SHOULD return a delta link, if
possible for such a request, but MAY ignore this value.
Query Options in Initial Query
Some query options, such as $filter, define the membership of a result, while other query options such
as $format, $top, and $skip, are used to control how the result is retrieved. The former are encoded into
the delta link while the latter are not, according to the following rules.
Query Options that Define the Result:
$filter is part of the definition of results. Services that support change tracking over filtered results must
logically include the $filter in the delta link.
Similarly, $select is part of the definition of results. If the initial query contained a $select clause, the
generated delta link should logically include the same $select, such that the delta query only returns
fields specified in the $select. However, services MUST NOT use the $select in determining which entries
have changed; that is, the entry should be returned whether or not the field that changed was in the
$select clause. This enables clients to know if any change has occurred to an entry, whether they
happen to have selected the changed field or not.
$expand may be specified on the initial query, in which case the delta link will return changes to the
expanded entities as well as relationships to the expanded entities, as described below.
$value is used to return a single value in raw form. Because delta links are applied to entities or sets of
entities, requests that include $value do not include a delta link.
$count is used to return the count of results. Because delta links are applied to a set of entities, $count
requests do not include a delta link.
Query Options that Affect How the Result is Retrieved:
Clients use $top and $skip to page through the results of a query. Delta Links represent changes to an
entire conceptual set of results, not just an individual page; thus the presence of $top and $skip in the
initial query does not affect the delta link returned by the request; the delta link will always represent
the full set of results.
Similarly, $inlinecount, if used in the initial request, is not encoded in the delta link.
The results of a delta query are ordered by the service in such a way as to guarantee consistency when
applied in the order provided. Therefore, specifying $orderby in the initial request does not affect the
delta link.
The delta link does not encode the request $format.
Delta Queries
The Client uses the delta link, possibly composing supported Query Operators as defined below, in order
to issue a Delta Query against the service.
Query Operators in Delta Queries
$filter and $select are part of the definition of results. Clients must not append $filter or $select to the
delta link.
Delta links return the set of entities and relationships according to the information encoded in the delta
link, as described below. $expand cannot be appended to a delta link.
The client can compose $top and $skip on top of the delta query, just as the initial query, in order to do
client-side paging of changes, and can compose $inlinecount in order to include a count of all entries
and deleted entries returned by the delta query. Added or deleted links are not included in the count.
Similarly, it is valid to compose /$count on to the path of a delta query in order to determine just the
number of entries added, changed, or deleted since an initial request. For hierarchical results, $top and
$skip apply to the root entities within the request; all related entities and added or deleted links for
those included root entities that have changed are included in the result.
Because the results of a delta query are ordered by the service in such a way as to guarantee
consistency when applied in the order provided, it is an error to add an $orderby clause to a delta
query.
A client can use $format to specify the format for changes, regardless of the format specified in the
initial request.
Because delta links return changes as a feed of entities, it is not valid to compose $value on to a delta
query.
$skiptoken provides an optional, opaque, service-defined token used to track state within a next link in
server-driven paging. Clients should never compose a $skiptoken on top of an initial or delta query as
the results of doing so are undefined.
Delta Responses
The url specified by the delta link can be used to retrieve the first page of changes. The format for the
response is the same as results from any other OData resource request, with the following exceptions:




Results from a delta query are always formatted as a feed and include deleted entries, link entries,
and deleted link entries, as well as inserted or changed records.
For changed entries, the result MUST include all changed fields (unless explicitly excluded through a
$select clause in the initial query), MAY contain additional unchanged fields from the record (or as
specified in $select), and MAY, but are not required to, include navigation links.
Entries MUST NOT include related entries (or their Entry IDs) inline. For a delta query that is tracking
changes to a set of related entities (for example, the result of a $expand request), the result is
represented as a single heterogeneous feed containing changed, added, and deleted entries as well
as added or deleted link entries. Related entries MUST specify the set with which they are
associated.
Results from a delta query MUST be ordered such that applying all changes, in order, is guaranteed
to lead to a consistent state. Generally this can be achieved by ordering the changes according to
when the change occurred, oldest to most recent.
Note that collections are treated as atomic values; any collections returned from a delta request must
contain all current values for that collection.
If more than one page of changes is available, each page contains a next link for the next page of
changes.
The final page of changes contains an odata delta link for subsequent changes. This delta link must
include all changes not seen by retrieving all pages of results from the previous delta link.
Although services should make every effort to return only changed records, it is legal to over-enumerate
changes (i.e., return a record for which the client will not see a change).
If the client attempts to use a URL from a delta link that is no longer valid (i.e., the service has garbage
collected deletions more recently than the delta link was generated, or the schema has changed such
that the request can no longer be satisfied) then the service responds with 410 Gone, and should
include the url in the location header for refetching the entire set.
Client-Driven Paging
In order to support client-driven paging of initial results (as well as delta results), delta Links are the
same across all client-driven pages (i.e., do not include $skip/$top). However, $skip and $top can be
applied to delta links in order for the client to page through the delta results, just as they would the
initial results.
As with server-driven paging, the delta-link will exist on the final page of results. Note that there may be
no changes in the final page if the $skip is equal to the number of changes available.
Expand Support
Clients that want to track changes to related sets of data do so by issuing an initial query that includes a
$expand of the relevant related entities. The delta link returned from such a query will monitor changes
to all related entities as well as added or deleted links between related entities.
Links that are created between two entities within the expanded result are returned through a linkentry specifying the id of the parent, the parent navigation property, and the id of the related object, in
the direction specified through $expand.
Links that are deleted between two entities in the expanded result are returned through a deleted-linkentry, specifying the id of the parent, the parent navigation property, and the id of the related object
being removed, in the direction explicitly specified through $expand.
A deleted-link-entry MAY, but is not required to, be returned for links from a deleted object to related
objects.
Additionally, if a linkEntry specifying a relationship with a max cardinality of 1 is present, it MAY, but is
not required to, be preceded with a deletedLinkEntry specifying the previous value related value.
Changes to related entities, as well as added or deleted links, appear in a single heterogeneous result
according to the format as described below.
Batching
Individual requests within a batch may return delta links as appropriate, and delta queries may be
batched just as any other request.
Client Interaction
Clients typically interact with a delta-enabled result in two stages; first populating a local store and then
using delta links to maintain that local store.
Initial Population
The initial population of the store may come through some out-of-band means (for example,
compressed data may be loaded from disk or tape, or shipped as part of an application).
In a common case, however, the initial population of the store is done through standard OData
requests, as shown below.
1. The client first issues an OData request for data, specifying odata-request-deltas in the Prefer
header. The service responds with the data (which the client stores).
2. If there is more than one page of results, the service includes a next page link, which the client
uses to retrieve (and store) subsequent pages of results.
3. On the final page, instead of a next link, the service returns a delta link that can be used to
retrieve changes from the returned results.
4. The client saves this delta link in order to request changes at a later point in time.
Result Maintenance
After establishing a base data set, the client uses the saved delta link to issue a delta query against the
service.
1. The client issues a Delta Query, using the stored Delta Link.
2. Results of the delta query are merged with stored results; existing records are updated, new
records are added, and records associated with deleted entries are removed.
3. If there are additional changes, the service includes a next page link in the delta query response.
The client continues to retrieve (and merge) subsequent pages of changes.
4. On the final page of changes, the service returns a new delta link that can be used to enumerate
changes subsequent to the last enumerated change.
5. The client saves this new delta link.
6. The client repeats this maintenance cycle using the new delta link at some interval based, for
example, on a schedule, out-of-band event or notification, or user interaction.
Format for Atom
This section describes representing delta links, deleted entries, added link entries, and deleted link
entries in the Atom format.
Related Entities in Atom
All changed or added entities, including related entities, are returned in the top-level <feed> element of
an ATOM payload; entities MUST NOT include related entities inline.
The atom:id element for related entities MUST include the metadata:set attribute to specify the
entityset of the related entity.
Delta Link in Atom
For Atom encodings, the delta link is exposed as a new feed level <link> element, i.e.:
<link rel=" http://odata.org/deltaLink" href=<urltoretrievedeltas> />
The Delta Link, like the next link, typically appears at the end of the link which technically violates the
atom spec. We are consistent.
Deleted Entries in Atom
Deleted entries are represented as a deleted-entry element as defined in The Atom "deleted-entry"
Element. The deleted-entry element includes the ref attribute containing the atom id of the deleted
resource and time the resource was deleted (which may be empty), and an optional odata-specific
attribute to specify whether the deleted entry represents an entry that was deleted (destroyed) versus
removed from membership in the result (i.e., due to a data change):
deletedEntry = element at:deleted-entry {
atomCommonAttributes,
attribute ref { atomUri },
attribute when { atomDateConstruct },
[,attribute metadata:reason { 'deleted' | 'changed' }]
}
Where metadata: is the metadata namespace for data services:
"http://docs.oasis-open.org/odata/ns/metadata/4.0"
Relationship Links in Atom
Added relationships are returned through a linkEntry element, defined in the OData Metadata
namespace. The linkEntry element includes a source attribute containing the atom id of the resource on
which the link was added, a relationship attribute representing the navigation property for which the
relationship was specified, and a target attribute containing the atom id of the related resource, as well
as an optional when attribute specifying when the link was created.
linkEntry = element metadata:link-entry {
attribute source { atomUri },
attribute relationship { text },
attribute target { atomUri }
[,attribute when { atomDateConstruct }]
}
Deleted relationships are returned through a deletedLinkEntry element, defined in the OData Metadata
namespace. The deletedLinkEntry element includes a source attribute containing the atom id of the
resource on which the link was deleted, a relationship attribute representing the navigation property for
which the relationship was deleted, and a target attribute containing the atom id of the related
resource, as well as an optional when attribute specifying when the link was deleted.
deletedLinkEntry = element metadata:deleted-link-entry {
attribute source { atomUri },
attribute relationship { text },
attribute target { atomUri }
[,attribute when { atomDateConstruct }]
}
Atom Delta Query Result Example
The example below shows the last page of changes returned by a delta query, formatted according to
the OData Atom format. This delta result represents the following ordered changes:


ContactName for customer 'BOTTM' was changed to "Susan Halvenstern"
Order 10643 was removed from customer 'ALFKI'




Order 10643 was added to Customer 'BOTTM'
Customer 'BOTTM' was set as the customer for Order 10643
The shipping information for order 10643 was updated
Customer 'ALFKI' was deleted
<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<feed xml:base="http://northwinddelta.cloudapp.net/DeltaService.svc/" xmlns:d="http://odata.org"
xmlns:m="http://odata.org/metadata" xmlns="http://www.w3.org/2005/Atom"
xmlns:at="http://purl.org/atompub/tombstones/1.0" >
<title type="text">Customers</title>
<id>http://DeltaService.svc/Customers</id>
<updated>2011-02-16T01:00:25Z</updated>
<link rel="self" title="Customers" href="Customers" />
<entry>
<id>http://DeltaService.svc/Customers('BOTTM')</id>
<title type="text" />
<updated>2011-02-16T01:00:25Z</updated>
<author><name /></author>
<link rel="edit" title="Customer" href="Customers('BOTTM')" />
<category term="NorthwindModel.Customer" scheme="http://odata.org/scheme" />
<content type="application/xml">
<m:properties>
<d:ContactName>Susan Halvenstern</d:ContactName>
</m:properties>
</content>
</entry>
<m:deleted-link-entry
source="http://DeltaService.svc/Customers('ALFKI')"
relationship="Orders"
target="http://DeltaService.svc/Orders(10643)"
when="2011-02-16T01:00:25Z"/>
<m:link-entry
source="http://DeltaService.svc/Customers('BOTTM')"
relationship="Orders"
target="http://DeltaService.svc/Orders(10643)"
when="2011-02-16T01:00:25Z"/>
<entry>
<id m:set="$metadata/Orders">http://DeltaService.svc/Orders('10643')</id>
<title type="text" />
<updated>2011-02-16T01:00:25Z</updated>
<author><name/></author>
<link rel="edit" title="Order" href="Orders('10643')" />
<category term="NorthwindModel.Order"
scheme="http://odata.org/scheme" />
<content type="application/xml">
<m:properties>
<d:ShipName>Bottom-Dollar Markets</d:ShipName>
<d:ShipAddress>23 Tsawassen Blvd.</d:ShipAddress>
<d:ShipCity>Tsawassen</d:ShipCity>
<d:ShipRegion>BC</d:ShipRegion>
<d:ShipPostalCode>T2F 8M4</d:ShipPostalCode>
<d:ShipCountry>Canada</d:ShipCountry>
</m:properties>
</content>
</entry>
<at:deleted-entry
ref="http://DeltaService.svc/Customers('ALFKI')"
m:type="Northwind.Customer"
when="2011-02-16T01:00:30Z"
m:reason="deleted" />
<link
rel="http://odata.org/deltaLink"
href="http://DeltaService.svc/Customers?$expand=orders&$deltatoken=8015" />
</feed>
Design in OData for JSON:
This section describes representing delta links, deleted entries, added link entries, and deleted link
entries in the JSON format.
MetadataUrl in JSON
The MetadataUrl for a delta result in JSON is the same as the metadata URL for the request, with
/@Delta appended to the end of the URL.
The set for related entries MUST be specified through the odata.set annotation, and MUST appear
before any property or property annotation.
Delta Link in JSON
Delta links in JSON are returned on the last page of results as a property on the feed object named
"odata.deltaLink". The value of the odata.deltaLink property is the URL used to retrieve the changes
from the current result.
Deleted Entries in JSON
Deleted Entries in JSON are returned as an object in the results array with an "odata.kind" value of
"deletedEntry". The deleted entry object has the following properties:
odata.kind – The odata.kind property MUST be the first property and MUST be "deletedEntry"
id – The id of the deleted element (same id as is returned when calling GET on resource)
when – An optional Datetime value indicating when the element was deleted.
reason – An optional string value indicated whether the element represents a resource that was
deleted (destroyed) versus removed from the result because it was changed.
Relationship Links in JSON
Added relationships are returned as an object in the results array with an "odata.kind" value of
"linkEntry". The linkEntry object has the following properties:
odata.kind – The odata.kind property MUST be the first property and MUST be "linkEntry"
source – The id of the object from which the relationship is defined
relationship - the name of the relationship property on the parent object
target – The id of the related object
created – An optional datetime value indicating when the link was created
Deleted relationships are returned as an object in the results array with an "odata.kind" value of
"deletedLinkEntry". The deletedLinkEntry has the following properties:
odata.kind – The odata.kind property MUST be the first property and MUST be
"deletedLinkEntry"
source – The id of the object from which the relationship was deleted
relationship - the name of the relationship property on the parent object
target – The id of the related object
deleted – An optional datetime value indicating when the link was deleted
JSON DeltaQuery Result Example
The example below shows the last page of changes returned by a delta query, formatted according to
the OData JSON format. This delta result represents the following ordered changes:






ContactName for customer 'BOTTM' was changed to "Susan Halvenstern"
Order 10643 was removed from customer 'ALFKI'
Order 10643 was added to Customer 'BOTTM'
Customer 'BOTTM' was set as the customer for Order 10643
The shipping information for order 10643 was updated
Customer 'ALFKI' was deleted
{
"odata.metadata":"http://DeltaService.svc/$metadata#Customers/@Delta ",
"value":
[
{
"odata.id":"http://DeltaService.svc/Customers('BOTTM')'",
"ContactName":"Susan Halvenstern"
},
{
"odata.kind" : "deletedLinkEntry",
"source":"http://DeltaService.svc/Customers(ALFKI)'",
"relationship":"Orders",
"target":"http://DeltaService.svc/Orders(10643)",
"when":"2012-11-07T15:38"
},
{
"odata.kind" : "linkEntry",
"source":"http://DeltaService.svc/Customers('BOTTM')",
"relationship":"Orders",
"target":"http://DeltaService.svc/Orders(10643)",
"when":"2012-11-07T15:38"
},
{
"odata.type" : "Northwind.Order",
"odata.id":"http://DeltaService.svc/Orders(10643)",
"odata.set":"#Orders",
"ShipName":"Bottom-Dollar Markets",
"ShipAddress":"23 Tsawassen Blvd.",
"ShipRegion":"BC",
"ShipPostalCode":"T2F 8M4",
"ShipCountry":"Canada"
},
{
"odata.kind":"deletedEntry",
"id":"http://DeltaService.svc/Customers('ALFKI')",
"when":"2012-11-07T15:38",
"reason": "deleted"
}
],
"odata.deltaLink":"http://DeltaService.svc/Customers?$expand=orders&$deltatoken=8015"
}
Issues:
1. Should we include a standard (optional) type property for deleteEntry, LinkEntry, and
deletedLinkEntry, or leave this specified through an annotation?
2. Should /$count and $inlinecount count just the added/changed/deleted entries, or include
added/deleted links?
3. Introduce "odata.kind" as a general annotation; default if omitted is "entry".
4. There may be cases where a service can provide changes but not tombstones. Is there any client
scenarios that would benefit from this clickstop?