So, super weird use case.
Basically, a client created objects and syncs them to the server. The server persists them, and returns that same object with a UID. When the client gets that UID object, it deletes the client version and saves the server version.
I’m worried that the client will send the object, and while the server is validating, disconnect. Then when the client sends the object again, we have duplicates.
I could generate a client ID to avoid his situation and persist that with the server object, but I was looking into a way to only persist objects if the client successfully receives the response, so we know it won’t resend the request
I googled around, but I couldn’t find anything. Is there a way to do this?
So, as I thought, my answer really demonstrates a lack of understanding on how HTTP works. I suspected that it wasn't possible with this technology - and it isn't - but there's really an underlying problem that I should have addressed.
The correct answer is to have an id generated on the client that is also stored in the database. The reason is because this makes the request idempotent - that is, the client can resend the same request as many times as it likes without messing up the server.
Whenever the server gets a request to make a new object, it simply checks our client ids sent. If that object already exists, don't make it again, just return the server generated object. Simple!
Related
I'm actually working on a Rails Application API. I have models with a lot of data.
I would like to be able from my client application to send like a Token, a timestamp or whatever to get the new, updated, deleted content/datas since the last request. While providing a new Token, timestamp for the futur request.
In that way, I have just to update my local cached content on the client side depending on the result of the request rather than update all of my local datas at each request.
After many researches on Google, I didn't find anything convincing.
I don't now how can I manage that on the server side ? Is that a good practice ?
If yes what's the best way to do it ?
I am working on CloudKit sync in my app ("Tiny data, all devices" model, with a custom zone in the private database).
CKModifyRecordsOperation contains clientChangeTokenData property of NSData type which is described in the docs as follows:
When you modify records from a fetch operation, specify a client-generated data token using this property to indicate which version of the record you last modified. Compare the data token you supplied to the data token in the next record fetch to confirm the server has successfully received the device’s last modify request.
I don't get why I should bother given that with each request, I get a completion block which tells me whether the server has successfully received my request. Why do I need to manually compare this client token?
Is specifying clientChangeTokenData required to handle my use case correctly? I track local data changes and push everything in the queue on each data change. Remote changes are tracked via zone subscription.
If it is required, how do I generate this token correctly given that I have all kinds of record changes in my CKModifyRecordsOperation (my API usage aims for batch operations). What is the general workflow here?
Thank you.
It's unclear from the docs so I'd guess the clientChangeTokenData is useful in the case of sending up a large modify records operation, e.g. deleting 100 records. Then say your app sends a fetch request in another operation with a query (or fetch changes) result set that would be affected by the modifications which either:
is started and is running concurrently to the existing modify operation which hasn't finished yet.
is started before the server has finished processing the results of the previous modifications (the docs tend to allude to a processing delay).
If the fetch completion contains a different clientChangeTokenData to the one sent with the modify then you know it hasn't received (or finished processing?) the changes yet. In this situation you could either error, with an alert to say the server needs more time, or automatically retry the fetch after some time.
By the way in my tests, this token is per-device.
You would only have a reason to check the token if you had local changes that you want to write to CloudKit and you want to make sure that your changes are based on the latest version of the data in CloudKit.
You could also just ignore the token and save the data anyway. If the data has changed in the mean time, you will get a CloudKit error and you could handle it then.
I'm starting an iOS app that consume a Restful API.
I have control over that API and I'm confusing with the caching policies.
To begin, I only need caching a concrete resoruce, but the problem is that resource can change when I insert new record in the database.
Then, how can I tell to the application "Hey! Make the request only if there have been changes and if not, you get the data from the cache!"
I'm using AFNetworking to make requests..
You'll have to make a decision on either server or client side and build your own protocol.
Example:
You could send the server JSON post request which contains the 'version' of the data you have in the app. On the server-side you will increment the version number each time the data gets refreshed. If the version number does not match at server-side, the server will respond with all new data, else the server responds JSON with 'up to date'
EDIT:
If you are looking for an HTTP response saying that the data is not modified. This is done on server side. You'll have to implement this in the server.
I would like to have an action that formats a response, responds to the client, and then commits something to a database only if the client managed to receive the response. From what I understand of after_filters, they only intercept the response before it is sent to the client. Any ideas?
As far as I know, there is no reliable way of confirming that the client received a message at the application level other than instructing the client to send an additional request, e.g. with javascript.
Instead of relying on the client to receive your 200 message, I would put a list of successful transactions with the API somewhere else in the API so that the client can proactively visit that list to verify that transactions went through.
I think this is a reasonable thing to do, despite the other commentator. I do agree with his approach to write javascript which sends an acknowledgement.
I wouldn't leave a transaction open that long, so I recommend saving the data with a tentative flag, and then when you get the acknowledgement, mark the data as acknowledged.
Let's assume we have a client/server interaction happening over unreliable network (packet drop). A client is calling server's RESTful api (over http over tcp):
issuing a POST to http://server.com/products
server is creating an object of "product" resource (persists it to a database, etc)
server is returning 201 Created with a Location header of "http://server.com/products/12345"
! TCP packet containing an http response gets dropped and eventually this leads to a tcp connection reset
I see the following problem: the client will never get an ID of a newly created resource yet the server will have a resource created.
Questions: Is this application level behavior or should framework take care of that? How should a web framework (and Rails in particular) handle a situation like that? Are there any articles/whitepapers on REST for this topic?
The client will receive an error when the server does not respond to the POST. The client would then normally re-issue the request as they assume that it has failed. Off the top of my head I can think of two approaches to this problem.
One is that the client can generate some kind of request identifier, such as a guid, which it includes in the request. If the server receives a POST request with a duplicate GUID then it can refuse it.
The other approach is to PUT instead of POST to create. If you cannot get the client to generate the URI then you can ask the server to provide a new URI with a GET and then do a PUT to that URI.
If you search for something like "make POST idempotent" you will probably find a bunch of other suggestions on how to do this.
If it isn't reasonable for duplicate resources to be created (e.g. products with identical titles, descriptions, etc.), then unique identifiers can be generated on the server which can be tracked against created resources to prevent duplicate requests from being processed. Unlike Darrel's suggestion of generating unique IDs on the client, this would also prevent separate users from creating duplicate resources (which you may or may not find desirable). Clients will be able to distinguish between "created" responses and "duplicate" responses by their response codes (201 and 303 respectively, in my example below).
Pseudocode for generating such an identifier — in this case, a hash of a canonical representation of the request:
func product_POST
// the canonical representation need not contain every field in
// the request, just those which contribute to its "identity"
tags = join sorted request.tags
canonical = join [request.name, request.maker, tags, request.desc]
id = hash canonical
if id in products
http303 products[id]
else
products[id] = create_product_from request
http201 products[id]
end
end
This ID may or may not be part of the created resources' URIs. Personally, I'd be inclined to track them separately — at the cost of an extra lookup table — if the URIs were going to be exposed to users, as hashes tend to be ugly and difficult for humans to remember.
In many cases, it also makes sense to "expire" these unique hashes after some time. For example, if you were to make a money transfer API, a user transferring the same amount of money to the same person a few minutes apart probably indicates that the client never received the "success" response. If a user transfers the same amount of money to the same person once a month, on the other hand, they're probably paying their rent. ;-)
The problem as you describe it boils down to avoiding what are called double-adds. As mentioned by others, you need to make your posts idempotent.
This can be easily implemented at the framework level. The framework can keep a cache of completed responses. The requests have to have a request unique so that any retries are treated as such, and not as new requests.
If the successful response gets lost on its way to the client, the client will retry with the same request unique, the server will then respond with its cached response.
You are left with durability of the cache, how long to keep responses, etc. One approach is to remove responses from the server cache after a given period of time, this will depend on your app domain and traffic and can be left as a configurable step on the framework piece. Another approach is to force the client to sent acknowledgements. The acks can be sent either as separate requests (note that these could be lost too), or as extra data piggy backed on real requests.
Although what I suggest is similar to what others suggest, I strongly encourage you to keep this layer of network resiliency to do only that, deal with drop requests/responses and not allow it to deal with duplicate resources from separate requests which is an application level task. Merging both pieces will mush all functionality and will not leave you with a clear separation of responsibilities.
Not an easy problem, but if you keep it clean you can make your app much more resilient to bad networks without introducing too much complexity.
And for some related experiences by others go here.
Good luck.
As the other responders have pointed out, the basic problem here is that the standard HTTP POST method is not idempotent like the other methods. There is an effort underway to establish a standard for an idempotent POST method known as Post-Once-Exactly, or POE.
Now I'm not saying that this is a perfect solution for everybody in the situation you describe, but if it is the case that you are writing both the server and the client, you may be able to leverage some of the ideas from POE. The draft is here: https://datatracker.ietf.org/doc/html/draft-nottingham-http-poe-00
It isn't a perfect solution, which is probably why it hasn't really taken off in the six years since the draft was submitted. Some of the problems, and some clever alternate options are discussed here:
http://tech.groups.yahoo.com/group/rest-discuss/message/7646
HTTP is a stateless protocol, meaning the server can't open an HTTP connection. All connections get initialized by the client. So you can't solve such an error on the server side.
The only solution I can think of: If you know, which client created the product, you can supply it the products it created, if it pulls that information. If the client never contacts you again, you won't be able to transmit information about the new product.