I have a server set up with Flask.
Everything works fine locally, HTTP requests can take longer than 60 seconds to resolve.
But when I deploy the server on Openshift, any request that takes longer than 60 seconds will time out automatically.
I have already changed the timeout on openshift to 10m, but that is not working. Any idea?
haproxy.router.openshift.io/timeout: 10m
The issue seems to be related to the way I set up OpenShift. I am using VPC environment which has its own connection timeout:
https://cloud.ibm.com/docs/vpc?topic=vpc-advanced-traffic-management#connection-timeouts
Related
We're using Contentful to manage CMS content. When you save content in Contentful it sends webhooks for a service we've set up on Cloud Run, which in turn ensures the updated content is built and deployed.
This setup has been previously so that the Cloud Run service was limited to 1 container max, with 80 concurrent requests limit. This should be plenty for the few webhooks we get occasionally.
Now when debugging complaints about content not being updated I bumped into a very persistent and irritating issue - Google Cloud Run does not try to process the 2 webhooks sent by Contentful, but instead responds to one of the 2 with status 429 and Rate exceeded. in response body.
This response does not come from our backend, I can see in the Cloud Run Logs tab the message generated by Google: The request was aborted because there was no available instance.
I've tried:
Increasing number of processes on the container from 1 to 2 - should not be necessary due to use of an async framework
Increasing number of containers from 1 to 2
The issue persists for the webhooks from Contentful.
If I try making requests from my local machine with hey that defaults to 200 requests with 50 concurrency, they all go through without any 429 status codes returned.
What is going on that generates 429 status codes when a specific client - in this case Contentful - makes ONLY 2 requests in quick succession? How do we disable or bypass this behavior?
gcloud run services describe <name> gives me these details of the deployment:
+ Service [redacted] in region europe-north1
URL: https://[redacted].a.run.app
Ingress: all
Traffic:
100% LATEST (currently [redacted])
Last updated on 2021-01-19T13:48:46.172388Z by [redacted]:
Revision [redacted]
Image: eu.gcr.io/[redacted]/[redacted]:c0a2e7a6-56d5-4f6f-b241-1dd9ed96dd30
Port: 8080
Memory: 256Mi
CPU: 1000m
Service account: [redacted]-compute#developer.gserviceaccount.com
Env vars:
WEB_CONCURRENCY 2
Concurrency: 80
Max Instances: 2
Timeout: 300s
This is more a speculation that an answer, but I would try re-deploying you Cloud Run service with min-instances set to 1 (or more).
Here is why.
In the Cloud Run troubleshooting docs they write (emphasis mine):
This error can also be caused by a sudden increase in traffic, a long container startup time or a long request processing time.
Your Cloud Run service receives webhook events from a CMS (Contentful). And, as you wrote, these updates are rather sporadic. So I think that your situation could be the same as the one described in this comment on Medium:
I tested “max-instances: 2” and the conclusion is I got 429 — Rate exceeded responses from Google frontend proxy because no container was running. It seems that a very low count of instances will deregister your service completely from the load-balancer until a second request was made.
If Google Cloud did indeed de-register your Cloud Run service completely because it was not receiving any traffic, re-deploying the service with at least one container instance could fix your issue. Another way would be to call your Cloud Run service every once in a while just to keep it "warm".
I'm debugging a issue for production.
Where the request takes more than 30s and it fails.
I'm trying to reproduce the issue on my local machine.
How can I set the request timeout to 30s in local.
Working on Ruby on Rails Application with puma.
You can use rack-timeout gem.
Best,
Théo
I have setup one Application load balancer crossed zoned. Initially all instances are in healthy status but after reboot a instance that particular instance shows unhealthy in the target group.
Protocol HTTP
Path /
Port traffic port
Healthy threshold 5
Unhealthy threshold 5
Timeout 5
Interval 30
Success codes 200
Are you sure your webservice has successfully started back up after you rebooted it?
You'll likely need to provide much more information (such as OS + webservice) if you are to get any proper answer, but as a general suggestion you your remote into your EC2 instance and figure out if the service (IIS, Apache, Nginx or otherwise) is actually running.
I run apache Webserver inside of a docker container.
To be able to use multiple php-versions, I use phpfarm inside of this docker-container.
After I configured xdebug and connect it to phpstorm, I wonder why the debug-session allways finishd with a 500 error in the Browser.
The timeout was nearly 40 - 50 Seconds after I request the Webpage.
Solution was to set the Timeout for the Server in the vhostfile for each php-version:
FcgidIOTimeout 300
With this parameter, the timeout isset to 300 Seconds.
Don't forget to restart or reload the Webserver.
I have a RoR app running on elastic beanstalk. I have occasionally seen 403 errors from Passenger for a while. Most of the time 1 server is running but this gets increased to 3 or 4 instances in busy periods during the day.
Session stickeyness is not turned on
I have noticed that when a new server is started the ELB is sending requests to it before bundle install has finished.
If I ssh to the newly started server I can see in /var/app/current/ that the app has not yet been installed and if I run top it looks like bundler is running and compiling things with cc1, etc.
/var/app/support/log/passenger.log shows that requests to valid urls within my rails app are being received and responded to with 404. Hardly surprising because the app isn't there yet
After 5-10 minutes all of the compiling is complete and the app files appear in /var/app/current and all is well.
This doesn't seem quite right to me. How do I set up the ELB / my rails app so that the ELB can tell when it is ready to receive requests?
I found the answer to this. There was no application health check url set. In this case the ELB pings the instance to see if it's healthy, i.e. it checks that it is booted rather than if rails is up and running. Setting the health check url to '/login/' fixed it for me because this gives a 404 until rails in running and a 200 afterwards.
Elastic beanstalk demands 2 correct responses before it deems an instance to be healthy. It checks the instance every 5 minutes. This means that an instance can take a while to start serving requests. i.e. it takes boot time + waiting for next poll from elb + 5 minutes before it sees any real traffic