I have a strange situation in the live environment and it is imposible to figure out what is the issue.
The users complain about the fact they are logged out after a random time.
On the client side it is an ASP MVC application that runs in IIS. Regarding the server side, it is and .net WebApi application that also runs in IIS. This is the session settings from web.config file of the client app:
<sessionState cookieless="false" mode="InProc" timeout="120" />
Even if I have set 120 minutes in config or in IIS advanced settings, they got the logout.
I have a recycle process set at 01:00 in the middle of the night. I have checked the logs files from EvenViewer and there are no other recycle processes trigered, only that one from the middle of the night.
The idea is that I can't reproduce it on my local machine. It happens at any time during the 120 minutes that are configured.
Few weeks ago, I have implemented a mechanism for 2FA using the following library: https://github.com/RobThree/TwoFactorAuth.Net and a functionality "Trust this browser for the next 30 days" in order to skip the MFA authentication if the user want this. Actually, I just keep and expiration date in a cookie in order to know when should I reset the option and ask again the user for a new authentication code. Don't know if it is related to this, but they told me that they encountered this situation after I released that 2FA implementation.
UPDATE After I read more articles in order so solve this issue, I want to add that the aplication is hosted in AZURE. Maybe this information helps.
UPDATE Advanced settings from IIS
UPDATE This are session state settings
Any suggestions would help. Thanks in advance.
I have an MVC 4 app and am using the default authentication provider. I'm not using persistent cookies.
I don't have any problems in development but when hosted at HostGator, I SOMETIMES get logged off when I try to create a new item (HTTP POST). When this happens, I end up at the log on page like I wasn't authenticated.
HostGator does NOT have the app on multiple web servers so I'm thinking I shouldn't have to worry about machinekey stuff. Am I wrong?
When this happens, I just log in again and create the item again and it will succeed. Once this happens, I can't recreate the issue. I try reopening the browser and even different browsers but creating items will always work. It only seems to happen again if I try much later.
Some additional info, the timeout is set to 2880 (the default for an MVC project), which I know is long but I can't see how it would be related. Still, thought I'd mention it.
So I can't look at IIS logs or event viewer to get any idea what could be happening but I can add more logging to the app. Can anyone provide ideas for what to check or what logging to add to diagnose?
Thanks
EDIT
I realized that I could get to the IIS logs so I compared the POST that succeeded and the one that failed and immediately noticed something.
When I first did the GET to load the Item/Create page/view, the cs-username was populated but when I did the POST to create the item, it was gone. I can see that when I logged in again and was able to successfully create the item, that POST did have the cs-username populated.
Why would it disappear between the GET and the POST? There was a 7 minute delay from the GET to the POST but I can see I logged on 1 minute before the GET so the session was only 8 minutes old when the post happened. I've double checked that I don't have sessionstate explicitly configured so the default should be 20 minutes. I feel like I'm onto something but not sure exactly what.
Might be worth adding Glimpse, although running that on deployed code is kinda risky. It would have the benefit, though, of letting you see what's actually happening on the server. I've never used HostGator, so I can't say for certain, but if they recycle app pools aggressively, that would invalid your login, and explain why the logoff seems to happen randomly.
We have recently moved our webserver to a new machine (IIS based) and in the asp.net mvc application I am working on, I have noticed that in the firebug console each request fails twice with a 401 error before finally succeeding on the 3rd try. This did not happen previously and the application seems to be running slightly slower than before, though that could just be my perception.
What might be causing this issue and what might I be able to do in order to avoid it?
I ran into a similar problem couple weeks back and the below post saved my day...
https://serverfault.com/questions/137073/401-unauthorized-on-server-2008-r2-iis-7-5
I hope this may help you...
Consider trying to check your IIS pool. Or maybe a timeout expiration?
I'm having a problem with a MVC (1.0) app that I can't figure out at all. There's two versions of the site (live and UAT) hosted on the same server. For each version of the site, the same code is shared by multiple organisations who each have their own database (MSSQL2005) and a separate web site in IIS (7.5) (pointed to the same code).
The UAT site has an update to the code and the database that is waiting to be deployed to the live site.
One of the customers ("customer A") is getting an error "104: Connection reset by peer" when they try to log in to the UAT site. They can see the login page but when they submit their login details the connection seems to be timing out (the requests seem to take ~130s to complete).
Customer A can log in fine to the live site. The other customers don't have a problem logging into the UAT site or the live site. If I try to log in as customer A, using their login details, it all works fine from within our network, and also from outside our network.
Customer A seems to be using squid as a proxy.
I can't think what the problem could be, and I've run out of ideas of things to test. The fact that I can log in as the customer fine and other customers don't have any issues seems to eliminate the code and database as problems.
What other things could I do to try and isolate the problem?
By dumping out the request data I was able to work out that something (I'm guessing the proxy) was removing the form values from the request. This obviously meant the app didn't work properly.
However, it seems whatever was removing the form data was leaving content-length unchanged, which would explain why the client was timing out waiting for more data and the server thought it was finished.
By using https instead of http (which we were going to do anyway), the request tampering seem to have stopped.
We're using EngineYard Cloud to deploy our Ruby on Rails application. We are running Rails v2.3.3.
EngineYard Cloud deploys to AWS instances in a manner similar to Capistrano. After each deploy, we're running into Invalid Authenticity Token errors. Specifically, any user that has previously visited our application and then visits after the deploy and then tries to submit a form gets an invalid authenticity token error. This error persists until they reset their cookies for the site. After they reset their cookies, the site works as expected with no errors.
We are using ActiveRecord's session store and sessions are being saved to the database.
This is the error we are seeing:
ActionController::InvalidAuthenticityToken
/usr/lib/ruby/gems/1.8/gems/actionpack-2.3.3/lib/action_controller/request_forgery_protection.rb:79:in `verify_authenticity_token'
The session object is nil after the deploy, however, the session data still persists in the database and the session ID cookie still exists:
Session:
session id: nil
data: nil
We haven't been able to explain this one. Any thoughts on what could be the root cause?
Thanks for any suggestions!
EDIT: Just to update on this, we've been able to isolate an example of the error.
1) User loads form
2) Code is updated on server
3) User submits form
** Invalid Authenticity Token error occurs
It seems that when the environment changes, Rails is unable to handle this with the authenticity token.
We've tried several steps to resolve:
Resetting the session
Deleting the session cookie (both in JavaScript and Rails)
Wiping the session table in the database after deploying code
Nothing works. The only thing that works is having the user clear their cookies client-side.
(We've been Googling (even tried Binging!) for answers, but no dice. This seems to be a similar related issue: http://railsforum.com/viewtopic.php?id=21479)
Also: initially we thought this was isolated to our deployment to EngineYard, but we've also been able to reproduce it on our development server that we deploy to via Capistrano.
Any thoughts would be gratefully accepted.
Thanks!
ANSWER: After extensive work by EngineYard (they're awesome!) they were able to diagnose the issue. The root cause of this issue is a bug with mongrel clusters. Mongrel doesn't seem to see the first post request after being started. EngineYard did extensive work to diagnose this:
There doesn't appear to be anything in your code causing the issue and I have found people outside of our environment that have experienced the bug as well (http://www.thought-scope.com/2009/07/mongrelcluster-rails-23x-bad-post.html). I suppose a lot of people don't see it because the first request to a site generally isn't a post or they chalk it up to flukes.
[There is a potential workaround using CURL.] The curl work around would do a simple GET request to each of your mongrels on the server to prime them so to speak. You could do this with capistrano, but that won't work if you deploy via the dashboard. You can find a short section on deploy hooks we have built into the infrastructure here:
https://cloud-support.engineyard.com/faqs/overview/getting-started-with-engine-yard-cloud
Adding a simple run curl http://localhost:500x > /dev/null should work (where x is the port you have 5000-50005 on your current setup).
We have addressed the issue by switching our stack from Mongrel to Passenger, but apparently, a fix for Mongrel is in the works. Hopefully, this helps someone who sees this same strange issue.
The authenticity token is a hidden field on the form that rails checks when the form is submitted to ensure that the post data is coming from a live session.
It is there as a security measure to prevent malicious people from using a form submit on their site to say a delete action on someones account.
You can turn it off on your whole app by adding this to config/environment.rb
config.action_controller.allow_forgery_protection = false
You can turn it off a single controller using
skip_before_filter :verify_authenticity_token
or turn it on
protect_from_forgery :except => :index
check out the ActionController::RequestForgeryProtection::ClassMethods docs for more details
It sounds like the secret key used for authentication is changing when you redeploy, invalidating all existing sessions.
Do you have the configuration parameter config.action_controller.session set anywhere, and if you do, is there anything which would cause it to change when you redeploy?
One of my apps has it configured in config/environment.rb, and a more recent one (generated with Rails 2.3) has it set in config/initializers/session_store.rb. The setting looks like:
config.action_controller.session = {
:secret => 'long-string-of-hex-digits'
}
If you don't have this configured for some reason, rake secret will generate a key for you, which can then be inserted into your configuration.
(If it is — and it's not being changed by your deployment processes — then I have no idea what's going on.)
If it would only be there for mongrels! I'm getting the exact same error on passenger as well (user loads form, deploy, submit -> invalid authenticity token). It'd be interesting to know how you solved the issue by switching to passenger? Any further hints are highly welcome. I'll have a closer look as well...
Cheers!
Have encountered this same problem with Rails 2.3 and a Mongrel cluster where the session secret is definitely set in the session initializer. The problem occured even after clearing the client cookies on the client.
However the suggestion of doing a curl get request across all the mongrels after they restart appears to work - thank goodness someone figured this out because it appears to be pretty darned obscure.
The only added info I can supply we are using Apache mod_proxy_balancer along with https in front of our Mongrels, however this problem was occuring before we turned on SSL. Is anyone seeing this with haproxy as the balancer instead of Apache?
This solved this issue for me :-) :-) :-)
https://rails.lighthouseapp.com/projects/8994-ruby-on-rails/tickets/4690-mongrel-doesnt-work-with-rails-238#ticket-4690-37 Posted by Mike Bethany
August 30th, 2010 # 06:43 PM.
I've never gone to any length to figure out the details, but for me, this is a client-side data rot issue. If I've been messing around with the way I store my sessions (and therefore, my authorization details,) I get this error from time to time. Clearing out the private browser data; cookies, authenticated sessions, the works, has always solved it for me.
Hope this helps.