We're running a rails app via heroku that connects to a windows azure VM, where I've set up a redis master/slave to act as a cache (slash quick reference data store). The problem is, we sporadically get redis timeouts. This is with a timeout of 10 seconds (which I know is more than it needs), and reestablishing redis connections on fork. And using hiredis as a driver.
Anyone have a clue why this might be happening? I know heroku and the azure vm are hosted on different coasts, so there's a bit of latency; could there be TCP request drops? I'm fairly out of ideas.
Related
I have a microservice that is taking in webhooks to process but it is currently getting pounded by the sender of said webhooks. Right now I am taking them and inserting the webhooks into the db for processing but the data is so bursty at times that I don't have enough bandwidth to manage the flood of requests and I cannot scale anymore as I'm out of db connections. The current thought is to just take the webhooks and throw them into a Kafka queue for processing; using Kafka I can scale up the number of frontend workers to whatever I need to handle the deluge of requests and I have the replayability of Kafka. By throwing the webhooks into Kafka, the frontend web server no longer needs a pool of db connections as it literally is just taking the request and throwing into the queue for processing. Does anyone have any knowledge on removing the db connectivity from Puma or have an alternative to do what's being asked?
Currently running
ruby 2.6.3
rails 6.0.1
puma 3.11
Ended up using Puma's before fork and on_worker_boot methods to not re-establish the database connection for those particular web workers within the config
I have been stuck on this issue for the past 3 days and unsure where to look now.
I have a simple Sidekiq implementation into my rails app.
I am working on: Rails 4.2.0, Sidekiq 4.1.2, Redis 3.0.6
The production app is running live with heroku, and I have 1 worker dyno and 1 web dyno.
The issue is this, and I am unsure on how to approach it or what I did to make it do this.
When I run the redis-cli on heroku I can see the clients that I have running. At most I have 2 or 3 clients running at any given time. I can easily kill the clients with
CLIENT KILL TYPE normal
So that's all fine and dandy. The part when things get a little tricky is when I fire up my server locally, and I am working in development. All of a sudden my redic-cli shows that I have 19 clients running. This will result in me logging
Err max clients reached
My assumption is that somehow locally I am directing sidekiq to work off the redis production url. I have to admit what I know about Redis and Sidekiq is limited, but I do have a basic understanding of how it should be working.
Any help or guidance would be appreciated.
Try using sidekiq -c 3 to limit your concurrency.
This ended up being a configuration error. Just in case anyone stumbles upon this question hopefully this will help them not overlook something like I did.
This issue was happening only when I was firing up my local server, so I knew it had something to do with me locally. I noticed that on my production redis:cli I was seeing clients that had my local IP in the ADDR column.
This led me to believe that my local machine was pushing clients to my production Redis server. Looking at my logs when I fired up my Procfile I saw the Redis url there so that only confirmed it.
Finally after searching through my code, I discovered that I had actually added the url into my .env, so when I fired up my server it was using that production Redis url. So I changed it to the appropriate IP address for local development on my .env file redis://127.0.0.1:6379 and everything is now working as normal.
I have a Rails 4.2 app running on Heroku. Occasionally there is an issue that causes most incoming requests to get a server error. For example, there could be a memory leak or a max database connection issue. How can I setup a script or service to automatically restart the server when it detects errors?
I think this service could ping the app every few minutes and if it detects an error, it should confirm there's really a problem and then run heroku restart. How could this be set up?
After Googling this topic, I came across Neptune.io, which seems to provide a useful service for this task.
I have a rails application hosted on Engine Yard Cloud. After deploy I am getting an error:
Process postgresql: POSTGRES_CONNECTION CRITICAL: DB 'postgres' could not connect to server: Connection refused Is the server running locally and accepting connections on Unix domain socket '/var/run/postgresql/.s.PGSQL.5432'
After this error the instance with the application become unreachable and reloading the instance is required. After reloading the instance I can deploy once, but with the next deploy I am getting the same error. I would appreciate any ideas on how to solve this.
The most likely cause of this condition is that the Postgres server has stopped responding. This most commonly occurs due to a low memory condition on Solo instances but can have other causes as well. The best way to explore this further is to file a ticket with the Engine Yard Support team and we can investigate this further for you.
Thank you,
Tyler
We're running a redis server in EC2 and to protect the connections established ssh tunnels from our production app machines to the redis instance. We used something like autossh -M 0 -f -NT -L6379:localhost:6379 -i /path/to/ec2.key root#ec2-xxxx.compute-1-amazonaws.com to establish persistent tunnels.
Now here's the problem, this seems to work about 90% of the time, but occasionally for whatever reason redis throws Redis::CannotConnectError: Error connecting to Redis on localhost:6379 (ECONNREFUSED) when initializing Resque from environment.rb. The given box already has numerous redis connections from other processes (unicorn & resque threads), so I know the tunnel is working. If I rerun the command it's almost always successful and is able to establish the connection without a hitch.
We're running Rails 4, Resque 1.24.1, Redis gem 3.0.5 and redis-server 2.6.12 on CentOS 6 on an m2-xlarge.
Any ideas? When this happens during deploy it's extremely frustrating as it kills capistrano, it's also causing sporadic failures in our nightly jobs that spin up their own rails environments.