I am writing a simple IMAP client that will be able to sync w/ any Google email account. I don't want to have to read the ENTIRE set of message headers on the server every time I sync in order to be assured that I do not miss something. I would prefer to not ever have to do that, and to rely on some field that ensures total order. For example, I would prefer to rely on Google extended Message ID field or even just on Receieved-Date and have my logic be: "keep reading backwards until you hit something you have previously read". But alas, it does not seem to be that simple.
What is the preferred way to do sync such that it is both efficient (in terms of time + bandwidth) and guaranteed (i.e., no missed messages)?
Thanks!
Related
I'm reading RFC 7162, trying to figure out how a client should behave when communicating with a server that supports CONDSTORE but not QRESYNC.
The initial connection case is simple enough: the client compares its cached modseq value with the HIGHESTMODSEQ response to the SELECT command, and if it is lower, the client can retrieve any changes with FETCH + CHANGEDSINCE or SEARCH + MODSEQ. After doing that, the client can store the HIGHESTMODSEQ value as its new cached modseq value for the mailbox.
However, if the client receives unsolicited FETCH responses while the mailbox is selected, can it draw any reliable conclusions from the MODSEQ attributes in these updates? Would it be safe to cache the highest seen MODSEQ value, or would the client risk missing any updates to the mailbox state in doing so?
This is tricky.
The question probably refers to a language in RFC7162 which clarifies that HIGHESTMODSEQ and MODSEQ are two different beasts, and that a FETCH MODSEQ might arrive while a server is forced to withdraw some EXPUNGEs, maybe because the client sent a command which uses MSNs (see motivation in chapter 3.2).
In this particular case where the server is said to only support CONDSTORE and not QRESYNC, the MODSEQ and HIGHESTMODSEQ are only used for keeping track of metadata changes (such as FLAGS updates). They do not have to change on message expunges. The situation which the RFC is trying to prevent because of QRESYNC therefore can happen without any fatal consequences as you as a client cannot use HIGHESTMODSEQ for any other purpose but FLAGS updates. Based on that, I don't see a reason why a client shouldn't bump HIGHESTMODSEQ on a CONDSTORE-only server upon seeing a MODSEQ.
You should probably ask on the imap-protocol mailing list and seek clarification. I, too, have an open bug about how to deal with this properly, and the RFC is not completely clear to me.
Let's assume we have a client/server interaction happening over unreliable network (packet drop). A client is calling server's RESTful api (over http over tcp):
issuing a POST to http://server.com/products
server is creating an object of "product" resource (persists it to a database, etc)
server is returning 201 Created with a Location header of "http://server.com/products/12345"
! TCP packet containing an http response gets dropped and eventually this leads to a tcp connection reset
I see the following problem: the client will never get an ID of a newly created resource yet the server will have a resource created.
Questions: Is this application level behavior or should framework take care of that? How should a web framework (and Rails in particular) handle a situation like that? Are there any articles/whitepapers on REST for this topic?
The client will receive an error when the server does not respond to the POST. The client would then normally re-issue the request as they assume that it has failed. Off the top of my head I can think of two approaches to this problem.
One is that the client can generate some kind of request identifier, such as a guid, which it includes in the request. If the server receives a POST request with a duplicate GUID then it can refuse it.
The other approach is to PUT instead of POST to create. If you cannot get the client to generate the URI then you can ask the server to provide a new URI with a GET and then do a PUT to that URI.
If you search for something like "make POST idempotent" you will probably find a bunch of other suggestions on how to do this.
If it isn't reasonable for duplicate resources to be created (e.g. products with identical titles, descriptions, etc.), then unique identifiers can be generated on the server which can be tracked against created resources to prevent duplicate requests from being processed. Unlike Darrel's suggestion of generating unique IDs on the client, this would also prevent separate users from creating duplicate resources (which you may or may not find desirable). Clients will be able to distinguish between "created" responses and "duplicate" responses by their response codes (201 and 303 respectively, in my example below).
Pseudocode for generating such an identifier — in this case, a hash of a canonical representation of the request:
func product_POST
// the canonical representation need not contain every field in
// the request, just those which contribute to its "identity"
tags = join sorted request.tags
canonical = join [request.name, request.maker, tags, request.desc]
id = hash canonical
if id in products
http303 products[id]
else
products[id] = create_product_from request
http201 products[id]
end
end
This ID may or may not be part of the created resources' URIs. Personally, I'd be inclined to track them separately — at the cost of an extra lookup table — if the URIs were going to be exposed to users, as hashes tend to be ugly and difficult for humans to remember.
In many cases, it also makes sense to "expire" these unique hashes after some time. For example, if you were to make a money transfer API, a user transferring the same amount of money to the same person a few minutes apart probably indicates that the client never received the "success" response. If a user transfers the same amount of money to the same person once a month, on the other hand, they're probably paying their rent. ;-)
The problem as you describe it boils down to avoiding what are called double-adds. As mentioned by others, you need to make your posts idempotent.
This can be easily implemented at the framework level. The framework can keep a cache of completed responses. The requests have to have a request unique so that any retries are treated as such, and not as new requests.
If the successful response gets lost on its way to the client, the client will retry with the same request unique, the server will then respond with its cached response.
You are left with durability of the cache, how long to keep responses, etc. One approach is to remove responses from the server cache after a given period of time, this will depend on your app domain and traffic and can be left as a configurable step on the framework piece. Another approach is to force the client to sent acknowledgements. The acks can be sent either as separate requests (note that these could be lost too), or as extra data piggy backed on real requests.
Although what I suggest is similar to what others suggest, I strongly encourage you to keep this layer of network resiliency to do only that, deal with drop requests/responses and not allow it to deal with duplicate resources from separate requests which is an application level task. Merging both pieces will mush all functionality and will not leave you with a clear separation of responsibilities.
Not an easy problem, but if you keep it clean you can make your app much more resilient to bad networks without introducing too much complexity.
And for some related experiences by others go here.
Good luck.
As the other responders have pointed out, the basic problem here is that the standard HTTP POST method is not idempotent like the other methods. There is an effort underway to establish a standard for an idempotent POST method known as Post-Once-Exactly, or POE.
Now I'm not saying that this is a perfect solution for everybody in the situation you describe, but if it is the case that you are writing both the server and the client, you may be able to leverage some of the ideas from POE. The draft is here: https://datatracker.ietf.org/doc/html/draft-nottingham-http-poe-00
It isn't a perfect solution, which is probably why it hasn't really taken off in the six years since the draft was submitted. Some of the problems, and some clever alternate options are discussed here:
http://tech.groups.yahoo.com/group/rest-discuss/message/7646
HTTP is a stateless protocol, meaning the server can't open an HTTP connection. All connections get initialized by the client. So you can't solve such an error on the server side.
The only solution I can think of: If you know, which client created the product, you can supply it the products it created, if it pulls that information. If the client never contacts you again, you won't be able to transmit information about the new product.
I'm interested in creating a challenge / response type process in Delphi. The scenario is this...we have 2 computers...1 belongs to the user and 1 belongs to a support technician.
The user is locked out of a certain program, and in order to gain 1 time access, I want:
The user to be presented with a challenge phrase, such as "28394LDJA9281DHQ" or some type of reasonably unique value
The user will call support staff and read this challenge (after the support staff has validated their identity)
The support person will type this challenge value into a program on their system which will generate a response, something equally as unique as the response, such as "9232KLSDF92SD"
The user types in the response and the program determines whether or not this is a valid response.
If it is, the user is granted 1 time access to the application.
Now, how to do this is my question? I will have 2 applications that will not have networked access to one another. Is there any functionality within Windows that can help me with this task?
I believe that I can use some functionality within CryptoAPI, but I really am not certain where to begin. I'd appreciate any help you could offer.
I would implement a MD5 based Challenge-Response authentication.
From wikipedia http://en.wikipedia.org/wiki/CRAM-MD5
Protocol
Challenge: In CRAM-MD5 authentication, the server first sends
a challenge string to the client.
Response: The client responds with a username followed by a space
character and then a 16-byte digest in
hexadecimal notation. The digest is
the output of HMAC-MD5 with the user's
password as the secret key, and the
server's original challenge as the
message.
Comparison: The server uses the same method to compute the expected
response. If the given response and
the expected response match then
authentication was successful.
This provides three important types of
security.
First, others cannot duplicate the hash without knowing the password.
This provides authentication.
Second, others cannot replay the hash—it is dependent on the
unpredictable challenge. This is
variously called freshness or replay
prevention.
Third, observers do not learn the password. This is called secrecy.
The two important features of this
protocol that provide these three
security benefits are the one-way hash
and the fresh random challenge.
Additionally, you may add some application-identification into the challenge string, for a double check on the sender of the challenge.
Important: it has some weaknesses, evaluate carefully how they may affect you.
Regarding the verbal challenge/response strategy: We used this approach to license a niche application on five thousand workstations world-wide for more than ten years. Our support team called it the "Missile Launch Codes" because of its similarity to the classic missile launch authentication process seen on old movies.
This is an extremely time consuming way to protect your program. It consumed enormous amounts of our staffs' and customers' time reading the codes to and from users. They all hated it.
Your situation/context may be different. Perhaps you won't be using it nearly as frequently as we did. But here are some suggestions:
Carefully consider the length and contents of the code: most users (and support staff) resent typing lots of characters. Many users are bad typists. Consider whether a long string and including punctuation marks and case sensitivity unduly burdens them compared to the amount of security added.
After years of using a verbal challenge/response implementation, we left it in place (as a fall-back) but added a simple automated system. We chose to use FTP rather than a more sophisticated web approach so that we didn't have to have any software running on our in-house server (or deal with our IT staff!)
Basically, we use FTP files to do the exchange that was previously done on the phone. The server places a file on the FTP server containing the challenge phrase. The file's name is the customer's name. Our support staff have a program that automatically creates this file on our ftp site.
The customer is instructed by our staff to hit a hot key that reads the FTP file, authenticates it, and places a response file back on the server.
Our support staffs' software has been polling waiting for the customer's software to create the response file. When it sees the file, it downloads it and confirms its contents, and deletes it from the server.
You can of course have this exchange happen as many times and in either direction as you need in a given session in order to accomplish your goals.
The data in the files can have the same MD5 keys that you would use verbally, so that it is as secure as you'd like.
A weakness in this system is that the user has to have FTP access. We've found that the majority of our users (all businesses) have FTP access available. (Of course, your customer base may not...) If our application in the field is unable to access our FTP site, it clearly announces the problem so that our customer can go to their IT staff to request that they open the access. Meanwhile, we just fall back to the verbal codes.
We used the plain vanilla Indy FTP tools with no problem.
No doubt there are some weaknesses in this approach (probably including some that we haven't thought of.) But, for our needs, it has been fantastic. Our support staff and customers love it.
Sorry if none of this is relevant to you. Hope this helps you some.
I'm trying to find my way around the OAuth spec, its requirements and any implementations I can find and, so far, it really seems like more trouble than its worth because I'm having trouble finding a single resource that pulls it all together. Or maybe it's just that I'm looking for something more specialized than most tutorials.
I have a set of existing APIs--some in Java, some in PHP--that I now need to secure and, for a number of reasons, OAuth seems like the right way to go. Unfortunately, my inability to track down the right resources to help me get a provider up and running is challenging that theory. Since most of this will be system-to-system API usage, I'll need to implement a 2-legged provider. With that in mind...
Does anyone know of any good tutorials for implementing a 2-legged OAuth provider with PHP?
Given that I have securable APIs in 2 languages, do I need to implement a provider in both or is there a way to create the provider as a "front controller" that I can funnel all requests through?
When securing PHP services, for example, do I have to secure each API individually by including the requisite provider resources on each?
Thanks for your help.
Rob, not sure where you landed on this but wanted to add my 2 cents in case anyone else ran across this question.
I more or less had the same question a few months ago and hearing about "OAuth" for the better part of a year. I was developing a REST API I needed to secure so I started reading about OAuth... and then my eyes started to roll backwards in my head.
I probably gave it a good solid day or 2 of skimming and reading until I decided, much like you, that OAuth was confusing garbage and just gave up on it.
So then I started researching ways to secure APIs in general and started to get a better grasp on ways to do that. The most popular way seemed to be sending requests to the API along with a checksum of the entire message (encoded with a secret that only you and the server know) that the server can use to decide if the message had been tampered with on it's way from the client, like so:
Client sends /user.json/123?showFriends=true&showStats=true&checksum=kjDSiuas98SD987ad
Server gets all that, looks up user "123" in database, loads his secret key and then (using the same method the client used) re-calculates it's OWN checksum given the request arguments.
If the server's generated checksum and the client's sent checksum match up, the request is OK and executed, if not, it is considered tampered with and rejected.
The checksum is called an HMAC and if you want a good example of this, it is what Amazon Web Services uses (they call the argument 'signature' not 'checksum' though).
So given that one of the key components of this to work is that the client and server have to generate the HMAC in the same fashion (otherwise they won't match), there have to be rules on HOW to combine all the arguments... then I suddenly understood all that "natural byte-ordering of parameters" crap from OAuth... it was just defining the rules for how to generate the signature because it needed to.
Another point is that every param you include in the HMAC generation is a value that then can't be tampered with when you send the request.
So if you just encode the URI stem as the signature, for example:
/user.json == askJdla9/kjdas+Askj2l8add
then the only thing in your message that cannot be tampered with is the URI, all of the arguments can be tampered with because they aren't part of the "checksum" value that the server will re-calculate.
Alternatively, even if you include EVERY param in the calculation, you still run the risk of "replay attacks" where a malicious middle man or evesdropped can intercept an API call and just keep resending it to the server over and over again.
You can fix that by adding a timestamp (always use UTC) in the HMAC calculation as well.
REMINDER: Since the server needs to calculate the same HMAC, you have to send along any value you use in the calculation EXCEPT YOUR SECRET KEY (OAuth calls it a consumer_secret I think). So if you add timestamp, make sure you send a timestamp param along with your request.
If you want to make the API secure from replay attacks, you can use a nonce value (it's a 1-time use value the server generates, gives to the client, the client uses it in the HMAC, sends back the request, the server confirms and then marks that nonce value as "used" in the DB and never lets another request use it again).
NOTE: 'nonce' are a really exact way to solve the "replay attack" problem -- timestamps are great, but because computers don't always have in-sync timestamp values, you have to allow an acceptable window on the server side of how "old" a request might be (say 10 mins, 30 mins, 1hr.... Amazon uses 15mins) before we accept or reject it. In this scenario your API is technically vulnerable during the entire window of time.
I think nonce values are great, but should only need to be used in APIs that are critical they keep their integrity. In my API, I didn't need it, but it would be trivial to add later if users demanded it... I would literally just need to add a "nonce" table in my DB, expose a new API to clients like:
/nonce.json
and then when they send that back to me in the HMAC calculation, I would need to check the DB to make sure it had never been used before and once used, mark it as such in the DB so if a request EVER came in again with that same nonce I would reject it.
Summary
Anyway, to make a long story short, everything I just described is basically what is known as "2-legged OAuth". There isn't that added step of flowing to the authority (Twitter, Facebook, Google, whatever) to authorize the client, that step is removed and instead the server implicitly trusts the client IF the HMAC's they are sending match up. That means the client has the right secret_key and is signing it's messages with it, so the server trusts it.
If you start looking around online, this seems to be the preferred method for securing API methods now-adays, or something like it. Amazon almost exactly uses this method except they use a slightly different combination method for their parameters before signing the whole thing to generate the HMAC.
If you are interested I wrote up this entire journey and thought-process as I was learning it. That might help provide a guided thinking tour of this process.
I would take a step back and think about what a properly authenticated client is going to be sending you.
Can you store the keys and credentials in a common database which is accessible from both sets of services, and just implement the OAuth provider in one language? When the user sends in a request to a service (PHP or Java) you then check against the common store. When the user is setting up the OAuth client then you do all of that through either a PHP or Java app (your preference), and store the credentials in the common DB.
There are some Oauth providers written in other languages that you might want to take a look at:
PHP - http://term.ie/oauth/example/ (see bottom of page)
Ruby - http://github.com/mojodna/sample-oauth-provider
.NET http://blog.bittercoder.com/PermaLink,guid,0d080a15-b412-48cf-b0d4-e842b25e3813.aspx
When a user completes an order at my online store, he gets an email confirmation.
Currently we're sending this email via Gmail (which we chose over sendmail for greater portability) after we authorize the user's credit card and before we show him a confirmation message (i.e., synchronously).
It's working fine in development, but I'm wondering if this will cause a problem in production. Will it require making the user wait too long? Will many simultaneous Gmail connections get us in trouble? Any other general caveats?
If sending the emails synchronously will be a problem, could someone recommend an asynchronous solution (is ar_mailer any good?)
The main issue I can think of is that Gmail limits the amount of email you can send daily, so if you get too many orders a day it might break.
As they say :
"In an effort to fight spam and
prevent abuse, Google will temporarily
disable your account if you send a
message to more than 500 recipients or
if you send a large number of
undeliverable messages. If you use a
POP or IMAP client (Microsoft Outlook
or Apple Mail, e.g.), you may only
send a message to 100 people at a
time. Your account should be
re-enabled within 24 hours. "
http://mail.google.com/support/bin/answer.py?hl=en&answer=22839
I would recommend using sendmail on your server in order to have greater control over what's going on and don't depend on another service, especially when sendmail is not really complicated to set up.
The internet is not as resilient as some people would have you believe, the link between you and GMail will break at some point or GMail will go offline causing the user to think that they have not paid sucessfully.
I would put some other queue in place, sendmail sounds acceptable and you can't create your site now for where it 'might' be hosted in the future.
Ryan
If the server waits for the email to be sent before giving the user any feedback, were there problems connecting to the mailserver (timeouts, server down etc) the user request would timeout too and he wouldn't be told anything about the status of his order, so I believe you should really do this asynchronously.
Also, you should check whether doing that is even allowed by GMail's TOS. If that's not the case, you may check if that's allowed if you purchase one of their subscriptions. Also, there's surely a limit to the number of outgoing emails you may send within a given timeframe so if you're expecting your online store to be successful, you may hit that limit and bump into some nasty issue. If you're not self-hosting the site, you should check whether your host offers email servers (several plans include them for free) as then using your host's ISP would be the most obvious choice.
FACT: Gmail crashes. Not often, but it happens, and you can't control it or test it.
The simplest quick-fix is to start a separate thread or fork a subprocess to send the email. Yes, there likely will arise problems from using Gmail, and I really have no input on that vs. the alternatives. But from a design perspective, there's just no reason to make the user wait for that process to complete.
From a testing perspective, this might be where a proxy pattern might come in handy. It might be easy for you to directly invoke Gmail to send a message. Make it harder. Put in a proxy object that does the mailing for you that you can turn off (because heaven knows you can't for testing purposes make Gmail crash). Just make your team follow what happens in the event of an email malfunction by turning off the proxy and trying to complete an order. If you are doing it synchronously, then all the plagues mentioned here by other posters will rear their heads. If you are doing it asynchronously, you should be able to allow it to fail silently (from the user's perspective--from your perspective there should be enormous logging statements and text messages in the middle of the night and possibly a mild electric current arcing across the surface of someone's skin).