We have a Spring 3 + REST application in which we are using a token to identify source of request coming in. This token is kept in-memory(in a hashmap) and used to identify the request.
When we are multiple instance of our app(deployed on multiple tomcat instances on different machines), how can we share/sync this token between different app nodes?
Our only requirement is to sync this token among different nodes.
I browsed and found few apis'/software like Redis, memcached, zookeeper. I am not able to decide which one to select.
Any kind of suggestion/comments is helpful.
Regards,
Pramod
I've never used zookeeper so I cannot say anything about it. Both Redis (database) and memcached (cache) will work for you. Which one is better depends on how you use tokens.
Choose Redis if:
tokens may be stored for a long time
no need to expire tokens
want to handle more tokens than can be stored in memory
want to replicate tokens to other servers so if one goes down other will provide tokens
tokens have to survive server restart
Choose memcached if:
tokens are valid only for certain amount of time and should expire
amount of tokens exeed memory capacity the least recently used token will be removed
all tokens may be stored in memory
no need to replicate tokens to other server
tokens don't have to survive server restart
want to use Spring Cache abstraction
Related
I have a access token that is stored encrypted as an ENV variable on a server. When someone wants to generate a report - they send up a password which is used to decrypt the encrypted access token (currently using ActiveSupport::MessageEncryptor) then the access token is used to make some API calls to a third party needed to generate the report.
The issue I'm running into is the network API call to the third party is slow (a few minutes) and causing web requests to timeout. The project has Sidekiq + ActiveJob integrated and I thought it'd be great to queue a job (then the slow runtime won't matter).
I'm unclear now how to properly handle the password. I can't use the password as an argument for the job (since it then gets stored in Redis). Likewise I can't put the decrypted access token (for the same reason). Do I have any options? If the third party let me convert the encrypted access key into a limited use / expiring key via a quick API I could do that prior to enqueueing - but I don't beleive that is an option. Any other options?
Sidekiq Enterprise supports encrypted job data, designed to solve exactly the problem you have.
https://github.com/mperham/sidekiq/wiki/Ent-Encryption
We have implemented our own oAuth provider and are having an issue when the system runs in a load balanced scenario. When we run with a single server all is well but when we switch the other on we get the following situation:
Token ‘A’ generated on server 1
Token ‘A’ not valid on server 2.
I have done some Googling on this and it seems to be a known issue but can’t seem to find a solution.
Anybody got an idea.
Thanks
You will have to make sure that you do one of:
synchronize the state of your Authorization Server between all load balanced nodes by using a shared cache (e.g. database or file system) or replicates state across nodes using some replication mechanism
your Authorization Server issues tokens that can be inspected by the load balancer to find out to which node it needs to send the validation request
The latter. has the downside that it cannot be used in a high availability scenario.
Struts 2 support stop double-submission of forms by generate a unique random token and store it in the session, and use token tag pass the token to the client form then verify the tokens from session and form.
As far as i know, this solution can only work in a single JVM because the session is separated from each other. I cannot find something useful about how to use this solution in a distributed Java environment. We use Nginx proxy HTTP requests to multiple JVMs and Nginx does not guarantee to proxy the same request to the same JVM every time.
Can someone give me some help?
BTW, i am trying to use this solution to stop CSRF attack.
You have two choices (neither of which really has anything to do with Struts 2, but has everything to do with session management in a distributed environment):
Use Session Affinity - so when a user creates a session, Nginx remembers which backend server the user went to, and that session is bound to that server for all subsequent requests. (This is the more typical solution). might get you started.
Depending on your application server, there may be the possibility of sharing the session data between servers. For example, in Tomcat 6, the configuration directions are.
If you are running a Rails 3 app with multiple web dynos on Heroku,
Every time you hit the app, do you typically connect with a different web dyno?
Can sessions work across different web dynos?
Does it work for different Rails session stores (ActionDispatch::Session::CookieStore,
ActiveRecord::SessionStore, and ActionDispatch::Session::CacheStore)
In short yes - sessions will work across multiple web dynos.
Sessions work across web dynos - because Rail's design of session support allows it to. If anything, the web dyno model is exactly how Rail's was intended to be scaled horizontally.
1. Every time you hit the app, do you typically connect with a different web dyno?
Based on heroku documentation:
The routing mesh is responsible for determining the location of your application’s web dynos within the dyno manifold and forwarding the HTTP request to one of these dynos. Dyno selection is performed using a random selection algorithm.
So dyno selection is random... but that dyno has to have your application installed. So if you have more than one dyno, then you may end up connecting to a different dyno (which is important as this facilitates load balancing and high availability)
2. Can sessions work across different web dynos?
Yes. Most web stacks support sessions by doing the following:
Assigning a session id - which is a unique id, and it is usually set as a session cookie so that the browser will always send the id with ANY HTTP request to the originating host
Providing storage which maps the session id to the actual session data
So by this process, sessions can be supported as every inbound HTTP request has the session ID, which is accessible by the web dyno when it handles your request.
3. Does it work for different Rails session stores (ActionDispatch::Session::CookieStore, ActiveRecord::SessionStore, and ActionDispatch::Session::CacheStore)
ActionDispatch::Session::CookieStore
Yes. The cookie store stores encrypted session data as a cookie. So your browser sends all the session data (encrypted) back to the host, which is then decrypted for use within your app.
ActiveRecord::SessionStore
Yes. The cookie store stores encrypted session data in a database table. An ID is then assigned as a cookie. So your browser sends the ID to the host, which is then used to load the session data from the database. Since all web dynos have a connection to the DB, this means it is also supported.
ActionDispatch::Session::CacheStore
Yes but you need a cache store service (eg MemCache addon). The cookie store stores encrypted session data in a cache store (memcache), which is a shared service across all web dynos. An ID is then assigned as a cookie. So your browser sends the ID to the host, which is then used to load session data from the cache store (memcache).
I do not believe Heroku makes any effort to send consecutive requests to the same web dyno. I might be wrong and they make some effort, but even if they do, it isn't likely to be anything like reliable enough to count on for session management.
However, ActionDispatch::Session::CookieStore will definitely work because the data is stored in an encrypted client-side cookie. ActiveRecord::SessionStore will work because the data is stored in the database, which is presumably shared by all web dynos. ActiveDispatch::Session::CacheStore should work if you use a MemCached server shared between all clients, or a similar shared cache.
The only thing that wouldn't work is some sort of file-based session storage on the local filesystem, and situations like multiple Heroku dynos is exactly why that type of session storage is not common in modern web applications.
I'm trying to find my way around the OAuth spec, its requirements and any implementations I can find and, so far, it really seems like more trouble than its worth because I'm having trouble finding a single resource that pulls it all together. Or maybe it's just that I'm looking for something more specialized than most tutorials.
I have a set of existing APIs--some in Java, some in PHP--that I now need to secure and, for a number of reasons, OAuth seems like the right way to go. Unfortunately, my inability to track down the right resources to help me get a provider up and running is challenging that theory. Since most of this will be system-to-system API usage, I'll need to implement a 2-legged provider. With that in mind...
Does anyone know of any good tutorials for implementing a 2-legged OAuth provider with PHP?
Given that I have securable APIs in 2 languages, do I need to implement a provider in both or is there a way to create the provider as a "front controller" that I can funnel all requests through?
When securing PHP services, for example, do I have to secure each API individually by including the requisite provider resources on each?
Thanks for your help.
Rob, not sure where you landed on this but wanted to add my 2 cents in case anyone else ran across this question.
I more or less had the same question a few months ago and hearing about "OAuth" for the better part of a year. I was developing a REST API I needed to secure so I started reading about OAuth... and then my eyes started to roll backwards in my head.
I probably gave it a good solid day or 2 of skimming and reading until I decided, much like you, that OAuth was confusing garbage and just gave up on it.
So then I started researching ways to secure APIs in general and started to get a better grasp on ways to do that. The most popular way seemed to be sending requests to the API along with a checksum of the entire message (encoded with a secret that only you and the server know) that the server can use to decide if the message had been tampered with on it's way from the client, like so:
Client sends /user.json/123?showFriends=true&showStats=true&checksum=kjDSiuas98SD987ad
Server gets all that, looks up user "123" in database, loads his secret key and then (using the same method the client used) re-calculates it's OWN checksum given the request arguments.
If the server's generated checksum and the client's sent checksum match up, the request is OK and executed, if not, it is considered tampered with and rejected.
The checksum is called an HMAC and if you want a good example of this, it is what Amazon Web Services uses (they call the argument 'signature' not 'checksum' though).
So given that one of the key components of this to work is that the client and server have to generate the HMAC in the same fashion (otherwise they won't match), there have to be rules on HOW to combine all the arguments... then I suddenly understood all that "natural byte-ordering of parameters" crap from OAuth... it was just defining the rules for how to generate the signature because it needed to.
Another point is that every param you include in the HMAC generation is a value that then can't be tampered with when you send the request.
So if you just encode the URI stem as the signature, for example:
/user.json == askJdla9/kjdas+Askj2l8add
then the only thing in your message that cannot be tampered with is the URI, all of the arguments can be tampered with because they aren't part of the "checksum" value that the server will re-calculate.
Alternatively, even if you include EVERY param in the calculation, you still run the risk of "replay attacks" where a malicious middle man or evesdropped can intercept an API call and just keep resending it to the server over and over again.
You can fix that by adding a timestamp (always use UTC) in the HMAC calculation as well.
REMINDER: Since the server needs to calculate the same HMAC, you have to send along any value you use in the calculation EXCEPT YOUR SECRET KEY (OAuth calls it a consumer_secret I think). So if you add timestamp, make sure you send a timestamp param along with your request.
If you want to make the API secure from replay attacks, you can use a nonce value (it's a 1-time use value the server generates, gives to the client, the client uses it in the HMAC, sends back the request, the server confirms and then marks that nonce value as "used" in the DB and never lets another request use it again).
NOTE: 'nonce' are a really exact way to solve the "replay attack" problem -- timestamps are great, but because computers don't always have in-sync timestamp values, you have to allow an acceptable window on the server side of how "old" a request might be (say 10 mins, 30 mins, 1hr.... Amazon uses 15mins) before we accept or reject it. In this scenario your API is technically vulnerable during the entire window of time.
I think nonce values are great, but should only need to be used in APIs that are critical they keep their integrity. In my API, I didn't need it, but it would be trivial to add later if users demanded it... I would literally just need to add a "nonce" table in my DB, expose a new API to clients like:
/nonce.json
and then when they send that back to me in the HMAC calculation, I would need to check the DB to make sure it had never been used before and once used, mark it as such in the DB so if a request EVER came in again with that same nonce I would reject it.
Summary
Anyway, to make a long story short, everything I just described is basically what is known as "2-legged OAuth". There isn't that added step of flowing to the authority (Twitter, Facebook, Google, whatever) to authorize the client, that step is removed and instead the server implicitly trusts the client IF the HMAC's they are sending match up. That means the client has the right secret_key and is signing it's messages with it, so the server trusts it.
If you start looking around online, this seems to be the preferred method for securing API methods now-adays, or something like it. Amazon almost exactly uses this method except they use a slightly different combination method for their parameters before signing the whole thing to generate the HMAC.
If you are interested I wrote up this entire journey and thought-process as I was learning it. That might help provide a guided thinking tour of this process.
I would take a step back and think about what a properly authenticated client is going to be sending you.
Can you store the keys and credentials in a common database which is accessible from both sets of services, and just implement the OAuth provider in one language? When the user sends in a request to a service (PHP or Java) you then check against the common store. When the user is setting up the OAuth client then you do all of that through either a PHP or Java app (your preference), and store the credentials in the common DB.
There are some Oauth providers written in other languages that you might want to take a look at:
PHP - http://term.ie/oauth/example/ (see bottom of page)
Ruby - http://github.com/mojodna/sample-oauth-provider
.NET http://blog.bittercoder.com/PermaLink,guid,0d080a15-b412-48cf-b0d4-e842b25e3813.aspx