Hi have a few NodeJS servers on GCP Run and when there is a lot of requests, Google adds more instances. I've noticed that when they are added, we temporarily get a lot of 503 errors but the adding of instances could be because of instances being added.
Things I have tried to do to fix this:
Reduce the concurrency from 1000 to 500 to 300. We are still getting 503 errors but in some cases fewer
Used health checks to ping an endpoint to make sure express is running before sending traffic. This also might have helped but we are still getting quite a few 503 errors.
I expect that we should be able to fix this so that we don't have any 503 errors but we are still getting quite a few.
What else should I try?
Update #1:
It takes about 10 to 20 seconds to start:
Update #2:
I noticed that the 503 errors happen in groups and that the requests are not just a few milliseconds which means that they are running when the 503 errors occur. That means that GCP Run is likely adding instances because of the 503 errors and the 503 errors are not happening when the instances are added.
What can cause a 503 error all of a sudden for an instance that is already running?
Related
I am running a Spring Cloud Gateway(version 3.0.3) Project with reactor netty(version 2.5.7). My system is with 8 core processor and 16 GB RAM. I have a route which calls the HttpBin API. When i launch load testing with Constant throughput of 25 tps, i am getting around 0.75% 502 (Bad Gate way) errors.
What might be the issue.
I tried increasing the Netty's worker Thread count to 200, but no change in the throughput. still getting around 0.75% 502 errors.
Update:- After upgrading Spring Cloud Gateway version to 4.0.0, the 502 error count is reduced. It is now 0.15% 502 errors.
ALB is giving me 502 errors randomly during the playwright test. When I debugged and checked ALB logs in Athena and I found elb_status_code is 502
Also access log entry says, the request_processing_time is 0.0, the target_processing_time is 0.005, and the response_processing_time is -1.
As per this AWS documentation:
https://aws.amazon.com/premiumsupport/knowledge-center/elb-alb-troubleshoot-502-errors/
If the elb_status_code is "502" and the target_status_code is "-", then your load balancer is the source of the HTTP 502 errors
But I didn't understand a way to fix it. Can someone please help me with it?
I had the same issue - which caused due to the keep-alive timeouts (the server used shorter timeouts than the load balancer)
Take a look on that post:
AWS Load Balancer 502
Sometimes the following error appears:
Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Additionally, a 503 Service Unavailable error was encountered while trying to use an ErrorDocument to handle the request.
How can I fix it?
In command prompt when I run bundle install
I got an error showing there
`fetch_http': bad response Service Unavailable 503
If anybody else faced such error before?
#ruby,
I suppose there will be some problem with your Internet connection. Please recheck your connection.
Or any plugin/gem is unfetchable.
503 ->Service Unavailable
That means:
Server cannot respond due to maintenance or overloading. For example, a servlet might return this header if some thread or database connection pool is currently full. Server can supply a Retry-After header.
the server was running fine the last few months then suddenly today and yesterday it just stop responding and returned the 503 error. it works when i restarted the server but in a few hours, it stops again and returns the 503 error. The problem is i don't know what causes the problem time to time. i checked the error log and the file size is too large; could this be the problem of the sudden error on the tomcat server?
Any Help Will do
Best to tail the log files, when the error occurs and debug from there. If the log files are too large, zero them out, make a backup of it, and restart Tomcat and try.