So, i have a big rails application that is using delayed_job to send emails and SMS to the users.
Once in a while, the delayed_job process will simply stop working without any message on the logs. I have finally pinpointed the problem as the delayed_job process crashs when it coulnd't connect to the database.
Is there any configuration i can make so it will retry the connection instead of just crashing? I've tried setting the reconnect: true on the database.yml file with no success.
Another option that i'm looking for is maybe using a monitoring tool like god or bluepill.
Try using the -m flag when you start delayed job - that should start a separate monitor process that in my experience has been very good about restarting the process.
Related
We are getting a lot of sentries issue with PG:ConnectionBad on Postgres in RDS AWS and Ruby on Rails
PG::ConnectionBad Sidekiq/BookingExtensionCheckWorker
could not translate host name “ls-XXXXXXXXXXXXfee44.XXXXXXXXX.eu-west-1.rds.amazonaws.com” to address: Name or service not known
In two weeks ago, we have migrate a new database and we change endpoint in the RoR files api
new database endpoint,
ls-XXXXXXXXXXXXXXXf3d4a.XXXXXXXX.eu-west-1.rds.amazonaws.com
It working fine with no issue between the new database and Ruby on Rails api. however, I get a lot of sentries issue said that sidekiq having an issue with connection database as they use old database address which it’s no longer used. I have to check sidekiq database; the code shows it’s connected to the new database. They keep go back to the old database when I run Ruby on rails.
Is there some way to find why sidekiq keep connection old database
Sidekiq is background service. So when someone deployed he has used some sort of like. Right now you are getting issue you think is fine but actually most jobs are not running. Which you will noticed in few days.
How can you check jobs If you sidekiq setup. Probably following url will lead to all jobs.
your_url/sidekiq
It will show you all jobs, I think you have option here to restart services. Just click to restart sidekiq. And everything would be fine.
How to start
As your sidekiq running with old configuration. So following steps could be dangrous, I think you must check how you have started your process. Otherwise following are some way people configure it and runt it.
systemctl restart sidekiq
If this does not work, check your command your deployment guy has setup some sort of scripts inside /etc/init.d folder
Some time developer use following simple line to run sidekiq
bundle exec sidekiq -d -P tmp/sidekiq.pid -L log/sidekiq.log
or
bundle exec sidekiqctl stop
I have a Rails 4.2 app running on Heroku. Occasionally there is an issue that causes most incoming requests to get a server error. For example, there could be a memory leak or a max database connection issue. How can I setup a script or service to automatically restart the server when it detects errors?
I think this service could ping the app every few minutes and if it detects an error, it should confirm there's really a problem and then run heroku restart. How could this be set up?
After Googling this topic, I came across Neptune.io, which seems to provide a useful service for this task.
At the moment, when I start my postgres server (after a configuration change, e.g.), my rails app goes bonkers with FATAL errors while the server is restarting.
Instead of throwing an error, i'd like my rails app to wait for 5 seconds for the box to come back up. Is there a way to make that happen?
I'm using Rails 4.1 and Postgres 9.3.
I added pgbouncer between my postgresql server and the app. pgbouncer allows me to pause connections while I restart the postgresql server and resume the connections when the server comes back up.
Problem: need to send e-mails from Rails asynchronously.
Environment: Windows 7, Ruby 2.0, Rails 4.1, Sidekiq, Redis
After setting everything up, starting Sidekiq and starting Redis, I can see the mail request queued to Redis through the monitor:
1414256204.699674 "exec"
1414256204.710675 "multi"
1414256204.710675 "sadd" "queues" "default"
1414256204.710675 "lpush" "queue:default" "{\"retry\":true,\"queue\":\"default\",\"class\":\"Sidekiq::Extensions::DelayedMailer\",\"args\":[\"---\\n- !ruby/class 'UserMailer'\\n- :async_reminder\\n- - 673\\n\"],\"jid\":\"d4024c0c219201e5d1649c54\",\"enqueued_at\":1414256204.709674}"
But the mailer method never seems to get executed. The mail doesn't get sent and none of the log messages show up.
How does Redis know to execute the job on the queue and does something else need to be setup in the environment for it to know where the application resides?
Is delayed_job a better solution?
I started redis in one window, bundle exec sidekiq in another window, and rails server in a third window.
How does an item on the redis queue get picked up and processed? Is sidekiq both putting things on the redis queue and checking to see if something was added that needs to be processed?
Redis is used just for storage. It stores jobs to be done. It does not execute anything. DelayedJob uses your database for job storage instead of Redis.
Rails process pushes new jobs to Redis.
Sidekiq process pops jobs from Redis and executes them.
In your MONITOR output, you should see LPUSH commands when Rails sends mail. You should also see BRPOP commands from Sidekiq.
You need to make sure that both Rails and Sidekiq processes use the same Redis server, database number, and namespace (if any). It's a frequent problem that they don't.
I'm trying to make Resque work with my project, but unfortunately it seems that for some reasons Resque is not able to write on Redis.
Redis seems to be configured correctly, I'm able to connect with redis-cli and issue commands, runs on port 6379 as configured inside my Rails 3.0.5 app.
When I try to Resque enqueue something the job is queued, but it doesn't seem that something actually happens on Redis (0 clients connected inside my Redis logs).
When I restart the console, the queue is empty, with no workers running.
Everything fails silently, I have nothing in my rails logs, nothing on the console, nothing if I start a worker, it just (obviously) doesn't find any job to perform.
https://gist.github.com/867620
Any suggestions on how to fix or debug this ?
The problem was that I was including resque_spec in the bundle.
Obviously, resque_spec was stubbing Resque.enqueue, making my mistake very stupid and very difficult to spot.