I am kind of getting lost with this one, we are using Tomcat7 on Windows 2008 server which is running behind IIS. The application framework is Struts2 and we have kept logs in the first and last line of a struts action method. Also, we have turned on tomcat access log valve. The problem,
1) The logs in the action class method states that the request enter and leave the method in a few milliseconds, post completing the entire processing.
2) The tomcat access valve logs show the same request taking more then a 10 minutes.
Where would the response be getting stuck in this case. No clue on where to start debugging, as this happens completely randomly.
Related
Am working with wso2esb4.9.0 and having around 160 services which are http and are processed frequently, Initially when the server is started every thing is fine all service request response is up to the mark,
After 10-12 days the ESB server gets hanged does not process any request and no exception are seen in the log file even,They are some request which may be piled up in the server and not allowing new connection to process.
when i do restart of the server all the connections get releases and works for other 10-12 days again.
But doing a restart of the server may not be a good idea to do , where can i find these connections and close them if possible and am i missing any config changes of wso2 esb.
Am trying to find some different connection number using JMX and also what to know if any one face this issue and found the possible solution.
I have an MVC 4 app and am using the default authentication provider. I'm not using persistent cookies.
I don't have any problems in development but when hosted at HostGator, I SOMETIMES get logged off when I try to create a new item (HTTP POST). When this happens, I end up at the log on page like I wasn't authenticated.
HostGator does NOT have the app on multiple web servers so I'm thinking I shouldn't have to worry about machinekey stuff. Am I wrong?
When this happens, I just log in again and create the item again and it will succeed. Once this happens, I can't recreate the issue. I try reopening the browser and even different browsers but creating items will always work. It only seems to happen again if I try much later.
Some additional info, the timeout is set to 2880 (the default for an MVC project), which I know is long but I can't see how it would be related. Still, thought I'd mention it.
So I can't look at IIS logs or event viewer to get any idea what could be happening but I can add more logging to the app. Can anyone provide ideas for what to check or what logging to add to diagnose?
Thanks
EDIT
I realized that I could get to the IIS logs so I compared the POST that succeeded and the one that failed and immediately noticed something.
When I first did the GET to load the Item/Create page/view, the cs-username was populated but when I did the POST to create the item, it was gone. I can see that when I logged in again and was able to successfully create the item, that POST did have the cs-username populated.
Why would it disappear between the GET and the POST? There was a 7 minute delay from the GET to the POST but I can see I logged on 1 minute before the GET so the session was only 8 minutes old when the post happened. I've double checked that I don't have sessionstate explicitly configured so the default should be 20 minutes. I feel like I'm onto something but not sure exactly what.
Might be worth adding Glimpse, although running that on deployed code is kinda risky. It would have the benefit, though, of letting you see what's actually happening on the server. I've never used HostGator, so I can't say for certain, but if they recycle app pools aggressively, that would invalid your login, and explain why the logoff seems to happen randomly.
We have a web application which does some computing and returns a file to the client. When the computing takes less then 5 minutes everything works fine on IE and Chrome and we get the file but if computing takes more than 5 minutes IE times-out with "Internet Explorer cannot display the webpage" message where as Chrome keeps running and eventually gets the file from the server.
I've tried changing registry settings like KeepAliveTimeout, ReceiveTimeout, ServerInfoTimeout of WinInet but it didn't help. Clicking the Diagnose Connection Problems button shows "Windows received an HTTPerror message: 403 (forbidden) from " message which I think is because it tries to again access the site without credentials and fails. When using fiddler the request terminates with 504 status and show this message "ReadResponse() failed: The server did not return a response for this request." Interestingly I've once observed that even Chrome times-out if Fiddler is running (haven't verified this by re-running though.)
This is an ASP.Net web application using MVC framework.
I've spent considerable amount of time but haven't been able to find a solution for this. Any useful pointers would really be appreciated.
From KB181050
You can usually break down long processes into smaller pieces. Or, the server can return status data to update users about the process. In addition, you can create a long server process that has a messages-based or asynchronous approach so that it returns immediately to the user after the job is submitted, and then notifies the user after the long process is finished.
In other words, create a <div> and fill that <div> immediately with a "processing" value after the request has been accepted by the server. Then, use ajax or javascript to update that same <div> with the result whenever you're finished with processing.
My situation is like this:
1. User uploads 150MB zip file, with 600 files inside. It takes 4 minutes or so to upload the file to the server.
2. Server processes the file contents, takes 70 seconds or so.
3. The server responds with Service Unavailable, with a log like, "could not forward the response to the client... stop button was clicked"
4. The Rails application log says, 200 OK response was returned.
So, I am guessing it must be a problem within one of Nginx or Passenger that is causing it return with the error even thought it is going fine inside the Rails app. My suspect is a timeout setting, because I could reproduce it by just putting a sleep of 180 seconds inside the long running method and doing nothing.
I will appreciate if you guys know what specific nginx/passenger config may fix it.
If you're using S3 as your storage you may consider using something like carrierwave_direct to skip passing the file through the web server and instead upload directly to S3.
Like noted above you could incorporate a queueing process like delayed_job.
https://github.com/dwilkie/carrierwave_direct
I presume that nginx is the public-facing server and it proxies requests through to another server running your RoR application for you. If this assumption is correct, you may need to increase the value of your nginx proxy_read_timeout setting for the specific locations that are causing you trouble.
For long run request, I think you should return an 'please wait' page immediately and make the processing background. After the processing is completed, set the task in the database as 'completed'. Within the period, whenever user refresh the page, return 'please wait' immediately. After completed, return the result. You can set an autorefresh timeout in the page to refresh the page after an estimated period.
I'd instantly store the upload somewhere and redirect to a "please wait" page which asks for the status of the background processing and could even display some progress bar then, e.g. using ajax.
For the actual background processing I'd recommend DelayedJob which worked great for us and supports easy job deployment and implementation.
I issue a simple GET request to my server, and it's coming back after ~1.2 seconds on average (using firebug NET tab, the "waiting for reqponse" part- not even the whole reponse time)
My ping to the server is 0.250
Using Passenger with rails 2.3.3, in the rails log the request is taking ~0.023
My server is on GoDaddy, so I checked their homepage with firebug also- the "waiting for reqponse" time for their page is ~0.320
Worst case should be around 0.4... so where did I lose the other 0.8 seconds?
What else can I check?
Edit:
Seems like it's unrelated to rails-
An image request (that only apache responds to, doest hit the rails at all) takes ~1.2 seconds also
GoDaddy may have a reverse-proxy between you and your HTTP server.
They may be doing something like sending you the response headers right away, then possibly serving you the contents of the response from cache.
So, from the standpoint of your HTTP server, the response is transmitted. Then it goes to GoDaddy's reverse-proxy, then finally to your web browser.
Try setting PassengerPoolIdleTime to 0 in your Servers or VHosts configuration.
Maybe your server is shutting down the application instances to fast and spawns a new instance with every request which usualy takes quite long.
Take a look at the documentation for more information on this setting:
http://modrails.com/documentation/Users%20guide%20Apache.html#PassengerPoolIdleTime
Where your files are hosted from for GoDaddy is not the same as where their homepage is hosted from.
Have you checked other pages you have hosted on the same server? Possibly due to database connections or "slow" connections like that can cause the page to take awhile before it's sent back to the client.
Doesn't sound like it is your problem, but the ISP's.
Can you do a wget to an internal ip/port to your rails app directly (or apache) from the same server?
That will tell you if the probaby is in the app stack or further upstream.
If you can, you can use apache tool, called ab "apache benchmark" to help.
The key is having a ssh access to your computer.