I'm running some Rails servers behind unicorn, which is behind nginx.
I can see a handful of POST requests with a status of 408 in nginx's access.log.
123.45.67.890 - - [17/Mar/2016:01:23:45 +1100] "POST /collections/ABC/items/DEFGH HTTP/1.1" 408 0 "http://catalog.paradisec.org.au/collections/ABC/items/DEFGH/edit" "MY FAVOURITE BROWSER"
But I can't see anything from the same time in unicorn.log (which doesn't log an awful lot of stuff) or production.log (which logs a fair amount of stuff).
Should Rails be logging anything in a HTTP 408 scenario?
production.log only logs requests which actually reach rails. It seems to me, that a 408 is caused by nginx. This can happen if your unicorn workers are busy. So in this case the request neither reached unicorn nor rails.
Related
We have an ALB load balancing 2 targets. But we are experiencing 502 responses from ALB quite often, that interrupts UI to throw an error. Our application is UI based running on node js and uses socket.io.
Sample log from ALB access log.
https 2019-06-10T09:29:46.987095Z app/DES-G1-USE4-ext-elb/7c8fddfc050d66f6 184.51.199.55:55418 10.72.72.155:8888 0.000 8.697 -1 502 - 876 485 "GET https://designer-use4.genesyscloud.com:443/socket.io/?EIO=3&transport=polling&t=Mj0kC0w HTTP/1.1" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:us-east-1:361190373704:targetgroup/DES-G1-USE4-tg-desg/4cc8feabb9ee8f89 "Root=1-5cfe2302-b19c6d5059c2fc6096e048e0" "-" "session-reused" 0 2019-06-10T09:29:38.289000Z "forward" "-" "-"
Here -1 502 - means that request is forwarded to the backend by ELB, but the target is not responded. And the connection is b/w ELB and target is closed somehow. As per https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html.
In our application log, there is no trace of this request reached our application.
Please help us to debug this issue from ALB.
Make sure that your application is responding with a 200-399 response code on the / path. The behavior that you are mentioning is often because the Load Balancer Health Check is not able to verify that your application is running correctly. Also, make sure that your server is able to respond on HTTP that path ( HTTP/2 is not yet supported ).
Potential causes of this:
firewall: make sure security groups of your application server and ELB are able to connect to each other on the designated port.
health check: the load balancer will kick an application server that crashes a few times. Make sure that the application server is attached and showing as healthy on the dashboard.
I am in the process of trying to set up an SVN repo using an apache web server. I was able to get the repo created and configured without too many problems. I can reach the repo via the browser, so I think the apache configuration is correct. The problem comes when I try to do the initial commit. When I run the commit command in the terminal, it hangs for several minutes before returning svn: E175012: Connection timed out. The initial commit is a single file, less than 100kb. Even more strange, after the command times out, it seems to create an HTTPd process on my system that uses 90% of the CPU.
I did some research to see if I could solve the problem myself, but so far nothing has worked. I was able to use Charles Proxy to monitor the HTTP requests and it looks like the svn client is sending the POST but it is never receiving a response from the server. After the default timeout (10 minutes) the client gives up and displays the timeout error.
I also tried setting up the repo using SvnServe instead of apache. I was able to read and write to the repo using svn://. However, the code I am working on expects to communicate with the repo over HTTP, so I still need to figure out what the problem is with apache.
Does anyone know what could be causing this issue? Are there any other steps I can take to troubleshoot the problem for myself?
[Update]
I checked the logs for my apache server. Here is what I'm seeing when I run the commit:
_myip_ - - [28/Feb/2017:10:04:04 -0500] "OPTIONS /my/repo HTTP/1.1" 200 190 "-" "SVN/1.9.5 (x86_64-apple-darwin16.1.0) serf/1.3.9"
_myip_ - - [28/Feb/2017:10:04:04 -0500] "OPTIONS /my/repo HTTP/1.1" 200 97 "-" "SVN/1.9.5 (x86_64-apple-darwin16.1.0) serf/1.3.9"
[Update 2]
In an attempt to further narrow down the cause of this issue, I tried setting up a different apache server in a Linux virtual machine. That server worked perfectly, and I was even able to read/write to it from osx. So it would seem that the issue is something specific to the apache server on OSX.
Please try this.
$ sudo chmod -R 775 /var/lib/svn
Reference URL-: https://gotechnies.com/setup-svn-server-ubuntu/
My Rails application is deployed on Amazon elastic beanstalk using Docker. Web requests flow into an nginx web server that forwards them to the thin rails server residing in docker. Somewhere along the way there's a bottleneck. Once every 50 requests (or so) I see nginx reports serving time which is x40 higher than the time the rails thin server reports.
here's an example:
NGINX (7490ms):
146.185.56.206 - silo [18/Mar/2015:13:35:55 +0000] "GET /needs/117.json?iPhone HTTP/1.1" 200 2114 "-" "Mozilla/5.0 (Macintosh;
Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/40.0.2214.115 Safari/537.36" 7.490 7.490 .
THIN (rails server): 171ms
2015-03-18 13:35:48.349 [INFO] method=GET path=/needs/117.json
format=json controller=needs action=show status=200
duration=171.96 view=109.73 db=29.20 time=2015-03-18 13:35:47
Can anyone supply some guidance how to troubleshoot such a situation? How do I find the source of the response time difference? I guess it could be either nginx or docker or thin or linux.
It sounds like one of the thin processes is under load doing a heavy task and nginx is still sending requests to the busy one. If there were a problem of queuing in Thin, the request would take a short time to be processed itself, but longer to get to the top of the queue. So first, check others requests before or around that request.
Second, if you are using upstream to serve (http://nginx.org/en/docs/http/ngx_http_upstream_module.html), apparently you could get among others $upstream_response_time and try to log it.
Third, you could also try to reproduce a similar setup in dev/qa and try a stress test. If you manage to reproduce it consistently, you could see number of request on each queue (i.e. http://comments.gmane.org/gmane.comp.lang.ruby.unicorn.general/848).
We have Ruby on Rails application, that is running on VPS. This night the nginx went down and responded with "502 Bad Gateway". Nginx error log contained lots of folowing messages:
2013/10/02 00:01:47 [error] 1136#0: *1 connect() to
unix:/app_directory/shared/sockets/unicorn.sock failed (111:
Connection refused) while connecting to upstream, client:
5.10.83.46, server: www.website.com, request: "GET /resource/206 HTTP/1.1", upstream:
"http://unix:/app_directory/shared/sockets/unicorn.sock:/resource/206",
host: "www.website.com"
These errors started suddenly, because previous error messages was 5 days earlier.
So the problem was in unicorn server. Then i opened unicorn error log and found there just some info messages, which doesn't connected with a problem. Production log was useless too.
I tried to restart server via service nginx restart, but it didn't help. Also there were not some pending processes of unicorn.
The problem was solved when i redeploy the application. And it is strange, because i deployed the same version of application 10 hours before server went down.
I'm looking for any suggestions how to prevent such 'magic' cases in future. Appreciate any help you can provide!
Looks like your unicorn server wasn't running when nginx tried to access it.
This can be caused by VPS restart, some exception in unicorn process, or killing of unicorn process due to low free memory. (IMHO VPS restart is the most possible reason)
Check unicorn by
ps aux | grep unicorn
Also you can check server uptime with
uptime
Then you can:
add script that would start unicorn on VPS boot
add it as service
run some monitoring process (like monit)
While doing a load test I found passenger throwing below error at first when lots of concurrent requests hit server. And, client side it gives 502 error code. However, after some requests say 1000- 2000 requests its works fine.
2013/07/23 11:22:46 [error] 14131#0: *50226 connect() to /tmp/passenger.1.0.14107/generation-
0/request failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.251.18.167, server: 10.*, request: "GET /home HTTP/1.0", upstream: "passenger:/tmp/passenger.1.0.14107/generation-0/request:", host: hostname
Server Details.
Passenger 4.0.10
ruby 1.9.3/2.0
Server Ec2 m1.xlarge
64-bit 4core 15gb
Ubuntu 12:24 LTS
Its a web server which servers dynamic webpages for rails framework
Can somebody suggest what the issue might be?
A "temporarily unavailable" error in that context means the socket backlog is full. That can happen if your app cannot handle your requests fast enough. What happens is that the queue grows and grows, until it's full, and then you start getting those errors. In the mean time your users' response times grow and grow until they get an error. This is probably an application-level problem so it's best to try starting there. Try figuring out why your app is slow, at which request it is slow, and fix that. Or maybe you need to scale to more servers.