Synchronization
This guide provides information and tips to create a bi-directional sync between CCP and your own data store. CCP and the underlying OData protocol provides a number of features to create a robust and reliable synchronization solution.
Prerequisites
This guide assumes the reader is aware and has basic knowledge of http, basic authentication, REST based communication and some knowledge of OData. It's also assumed that the user has access to CCP. When the reader isn't aware of the before mentioned things it's highly recommended to read the Getting Started, Security Overview and Basic OData guides.
HTTP Requirements
CCP exposes a HTTP endpoint to communicate with. It conforms to the normal REST based pattern of using the HTTP methods GET for retrieving data, POST for creating data, PUT/PATCH for updating data and DELETE for removing data. It uses the OData standard for this. See http://www.odata.org for more information about the standard and to acquire basic knowledge of communicating using OData. CCP currently uses v3.0 of the standard.
On top of the OData standard, CCP requires some additional things to successfully make you're first request. This has to do with the security system of CCP and the versioning system of the API.
Every request to the CCP endpoint has to be authenticated using a provided username and password. CCP uses Basic Authentication for this.
Syncable Entities
Most entities in CCP are designed to be syncable by a clientapplication. A syncable entity can be recognized by 2 properties that are available, Deleted and LastUpdated. With these properties a client has all the information to sync two datastores without requesting every record on every sync.
NOTE: When using the metadata to generate entity code automatically you can use an included ValueAnnotation to recognize syncable entities. The Term Attributes.Syncable is added to the entity annotations with a Bool value of “true”.
ReadOnly Entities
A number of entities in CCP are read only. Those entities are mostly summaries of data from a number of other entities and are there for convenience. Trying to write data to a read only entity results in a ChangeReadOnlyResourceException exception.
NOTE: When using the metadata to generate the entity code automatically you can recognize read only entities using an annotation. For the entity there is a ValueAnnotation with the Term Attributes.ReadOnly and a Bool value of “true”. Read only is also reflected in the entityrights where all rights except reading is set to false. It's recommended to use the annotation because read only and not allowed to write are technically two different things.
Headers
Besides the http and OData specific headers, CCP also has headers. The following headers are available.
| Name | Possible values | Description |
|---|---|---|
| CabmanCloudPlatformVersion | Positive Integer value or a decimal “major.minor” | With this header the client requests data for a specific version of the interface. This ensures that newer versions of the interface do not conflict with the client. It is recommended that developers periodically check the docs and version information for new versions which could contain functionality that is needed. NOTE: When the requested version is not available anymore, the first available version is used but only when it is lesser than or equal to MaxCabmanCloudPlatformVersion |
| MaxCabmanCloudPlatformVersion | Positive Integer value or a decimal “major.minor” | With this header then client informs the server what the maximum version is that it supports. The server will never return a version higher than provided in this header. |
| ShowDeleted | 0 or not include header in request = Do not include logically deleted records in results 1 = include logically deleted records in results |
When a client adds a ShowDeleted header with value 1 it notifies the server that it can handle logically deleted records in the results. This is especially useful when syncing data stores. |
| ShowArchived | 0 or not include header in request = Do not include logically archived records in results 1 = include logically archived records in results |
When a client adds a ShowArchived header with value 1 it notifies the server that it can handle logically archived records in the results. This is especially useful when syncing data stores. |
The CCP versioning headers are considered mandatory on every request for every client connection to CCP. The versioning headers ensure that a client keeps working when the interface changes and the client can't handle these changes. When no versioning headers are provided, the server assumes the client can handle the latest version.
NOTE: An application is not forced to use a single version. On every request an application can request a different version. If an application needs a specific version of an entity but isn't updated for other entities it's possible to request a higher version for the supported entity but a lower version for the rest. It is not a recommended practice and be aware of dependencies but it is possible.
NOTE: The metadata contains Annotations with the minimum, maximum and current version of the CCP interface. The Annotation with the Target named “System.Data.Objects.CCPService” contains three ValueAnnotations with the following terms
- ServiceVersion.Min contains a Decimal with the lowest supported version
- ServiceVersion.Max contains a Decimal the highest supported version
- ServiceVersion.Current contains a Decimal with the version at which the metadata was requested.
Logically Deleted/Archived
A very tricky concept in a multi-tenant system like CCP is deletion. When multiple clients have access to the same record and one client deletes/archives the record, the other clients need to be made aware of the deletion. CCP solves this problem by not allowing permanent deletion of records that are sync able. When a client is allowed to delete a record (see EntityRights section for more information) and deletes it using a HTTP delete this is denied. When deleting a record the client must use a PUT operation and update the Deleted property of the record with the value True. In combination with the ShowDeleted/ShowArchived header other clients can still see the deleted record and process it in their own datastore.
Recommended practices for synchronization
This paragraph provides you with some tips to efficiently implement a 2-way sync with CCP and your own data-store.
The data-store can grow a lot over time. You don't want to request every piece of data every single time. You only want data that's changed since the last time it was requested.
Every syncable entity has a LastUpdated property which is a timestamp with a millisecond accuracy. This property is updated by the server on every change. When syncing entities with your local data-store you need to store this LastUpdated property to know if the entity has been changed by another user or application. When updating a record you can use a similar approach. You can request the lastupdated property of the entity you want to modify. If it has changed you don't have the most recent version. This is a passive approach to maintaining concurrency. The client is responsible for conflict resolution.
The OData protocol describes a similar system using standard ETags in combination with the “If-Match” and “If-None-Match” headers. By adding the ETag you received with the last update or create you signal the server to check if the ETag corresponds with the entity and if it doesn't match/or matches (depending on the “if” header) return a http 412 error and cancels the request. This is a more active approach to maintaining concurrency.
To prevent requesting every single entity every single time you can implement a sliding window approach using the LastUpdated property. The following steps are recommended to minimize the number and size of the requests/responses but still synchronizing every single change.
On first sync you do not have a window so the only way is to request everything. Performing a GET on the EntitySet (e.g. /Travelers) returns every single entity visible to the user. It's important to sort the results by LastUpdated so that the most recent LastUpdated is the last entity in the list. For sliding windows you need the most recent timestamp. With OData it's possible to request the entities in a specific order. You can do this using the $orderby query option. Append the query option to the request URI. (e.g. /Travelers?$orderby=LastUpdated)
For every single entity you need to store the LastUpdated to know if the record has changed. For the sliding window method you need to store the most recent LastUpdated on an EntitySet level. This way, the next time you request the EntitySet, you can use the LastUpdated to request every record that's updated after that timestamp. This drastically reduces network traffic and processing by the client application.
NOTE: There is one gotcha in this approach, Latency. There is network latency from request until response. CCP is a high-traffic, multi-threaded system so it's perfectly possible that between the request and the response an entity is updated. If you don't want to miss a single update it's recommended to modify your window to a few seconds earlier than the stored LastUpdated. This way you'll get these intermediate updates in the next request.
By checking every LastUpdated with the one in the response you can choose to only update your local data-store when the timestamps do not match. This optimistic approach to concurrency ensures that the end-user doesn't have to be bothered with conflict resolution questions.
Storing entity identification
When synchronizing two different data-stores there is always the case of record identification. Every entity stored in CCP has its own unique identification. The reference documentation show these properties in every EntitySet with a key icon next to it. Use this property to link your own identification to the CCP entity.
Recommended workflow
The following workflow is a recommended practice for 2-way synchronization of CCP entities.
- Periodically send changes from your software to CCP.
- Check if the entity is changed since the last update by checking LastUpdated first and use ETags when available
- Try to minimize requests for entities that don't update that often. This reduces network traffic considerably and frees up bandwidth to request entities that do change a lot.
- Update CCP when you have the latest version, update from CCP if it has a newer version.
- Only update properties that have changed
- Perform silent conflict resolution as much as possible.