Recently I just changed server for my production environment from passenger to puma. Before puma delayed job was working fine without any error but after puma it stopped working.
I am running 3 Delayed job to handle incoming requests on two different servers by dividing them as server 1 : 1 Job and server 2 : 2 Jobs.
RAILS_ENV=production bin/delayed_job -i first --queues=job_1,job_2 start
This command I am using to start delayed job after deployment. I am using Capistrano 3 for deployment. Also I have upgrade version of Capistrano version 2 to 3.
In Capistrano 2 we had
require 'delayed/recepies'
but as per delayed job official documentation I have created a rake task and executing in order to start job again.
It creates a new entry in delayed_jobs table as Delayed::Backend::ActiveRecord::Job.
What's wrong I am doing here? Is there any worker not listening to my incoming calls? Should I need to start additional things?
----UPDATED----
This is the rake task I have created to restart the jobs.
task :restart do
on roles(delayed_job_roles) do
within release_path do
with rails_env: fetch(:rails_env) do
execute('cd #{current_path};RAILS_ENV=production bin/delayed_job -i first --queues=job_1,job_2 stop')
execute('cd #{current_path};RAILS_ENV=production bin/delayed_job -i second --queues=job_1,job_2 start')
end
end
end
end
after 'deploy:publishing', 'delayed_job:restart'
As I have mentioned that I am running two seperate servers to manage delayed_jobs. On server 2 delayed_job.log file is not updated with latest logs. When I try to run delayed_job it executes on server 1 instead of server 2. Queues are running over server 2 to handle those jobs and when it runs on server 1 it couldn't find the exact source, which I am looking for to get executed.
Related
I want to run both the database and migration in foreman. However, I found that they usually run at the same time when I start foreman. Since at the time I run migration the database has not fully started yet, it causes the migration to fail.
Heroku using Procfile could facilitate a release phase. The phase would be run after all the commands are run. Can I do the same using foreman in my computer?
Heroku does not rely to Procfile to maintain the release process. The Heroku build stack does.
Since the foreman provides us the way to run multiple processes at the same time, not running processes in order, so your problem is not the responsibility of foreman
However, you have some other ways to do so.
Simple: since foreman can start your process with shell command, you can use basic shell command sleep (in seconds) for delaying your process
db_process: start_db_script.sh
migrarion_process: sleep 5; bundle exec rake db:migrate --trace
Full control: Instead of run default migration rake task, you can write another rake task which check the connection to database before execute the migration ( refer to this answer)
retried = 0
begin
# Establishes connection
ActiveRecord::Base.establish_connection
# Try to reconnect
# It will raise error if cannot reach your database
ActiveRecord::Base.connection.reconnect!
Rake::Task["db:migrate"].invoke if ActiveRecord::Base.connected?
rescue => e
retried += 1
if retried <= 5 # Retry only 5 times
sleep 1 # Wait 1 seconds before retry
retry
end
puts "#{e} Cannot connect to your database with 5 seconds"
end
I have a a rails app that is using Rufus Scheduler. When I turn on the rails server with:
rails s --port=4000
Rufus scheduler runs its tasks. If I run the rails server with:
rails s --port=4000 --daemon
Rufus no longer does its tasks. I added a couple of log messages. Here is the schedule code:
class AtTaskScheduler
def self.start
scheduler = Rufus::Scheduler.new
p "Starting Attask scheduler"
scheduler.every('5m') do
# test sending hip chat message
issue = Issue.new
issue.post_to_hipchat("Starting sync with AtTask","SYNC")
p "Launching Sync"
Issue.synchronize
end
end
end
Hipchat never gets the message from the scheduler and the log never gets the statement "Launching Sync".
Any ideas on what may be causing this?
There is documentation of this issue in the rufus-scheduler docs:
There is the handy rails server -d that starts a development Rails as
a daemon. The annoying thing is that the scheduler as seen above is
started in the main process that then gets forked and daemonized. The
rufus-scheduler thread (and any other thread) gets lost, no scheduling
happens.
I avoid running -d in development mode and bother about daemonizing
only for production deployment.
These are two well crafted articles on process daemonization, please
read them:
http://www.mikeperham.com/2014/09/22/dont-daemonize-your-daemons/
http://www.mikeperham.com/2014/07/07/use-runit/
If anyway, you need something like rails server -d, why not try bundle exec unicorn -D
instead? In my (limited) experience, it worked out of the box (well,
had to add gem 'unicorn' to Gemfile first).
For production purposes I need three processes running. This is my procfile and I use Foreman to start them:
web: bundle exec rails s Puma -p $PORT
queuing: bundle exec clockwork clock.rb
workers: bundle exec rake resque:workers
For deployment I'm using Mina. What's the appropriate way to start Foreman at the end of deploy task? Currently I'm starting like this:
desc "Deploys the current version to the server."
task :deploy => :environment do
deploy do
invoke :'git:clone'
invoke :'deploy:link_shared_paths'
invoke :'bundle:install'
invoke :'rails:db_migrate'
invoke :'rails:assets_precompile'
to :launch do
queue "touch #{deploy_to}/tmp/restart.txt"
queue "bundle exec foreman start"
end
end
end
... but I don't think that's the proper way since the "mina deploy" command never successfully exits and the local console just starts outputting whatever these processes are doing.
Question number two: How do I initialize logging for each of these three processes separately in separate files?
And how do I prevent killing all of these three processes when one of them crashes? How do I make the process restart when it crashes?
Thanks!
OK, so that's 3 questions.
1) I think you want to detach foreman process from the terminal. That way the deployment process will finish and foreman process will be running even after you have disconnected from the server. nohup is great for that, e.g. this will launch your app and pipe all logs to server.log file:
nohup foreman start > server.log 2>&1 &
2) AFAIK, foreman doesn't let you do that. You should probably use another process management service (e.g. systemd, upstart). Thankfully, foreman lets you easily export your config to different process management formats (http://ddollar.github.io/foreman/#EXPORTING).
3) Again, you probably want to separate your processes and manage them separately via upstart, systemd, etc.
I have my rails site deployed under apache. The apache is run as a service. Now I have added delayed_job there and things work fine.
Now I want to start the workers together with apache, e.g, After rebooting the server, my site and workers are up and ready so I don't have to log in and type "sudo RAILS_ENV=production script/delayed_job -n 2 start".
Another issue is that whenever I want to start the delayed_job I have to use "sudo"...
Any idea how to avoid those 2 issues?
Thanks for your help.
Use the whenever gem and its 'every :reboot' functionality. In schedule.rb:
environment = ENV['RAILS_ENV'] || 'production'
every :reboot do
command "cd #{path} && #{environment_variable}=#{environment} bin/delayed_job --pool=queue1:2, --pool=queue2,queue3:1 restart"
end
Could you just create a shell script to execute the commands you need?
#!/bin/sh
# stop delayed job
# restart apache
apachectl restart
# start delayed job
sudo RAILS_ENV=production script/delayed_job -n 2 start
It sounds like you want to have delayed_job automatically start after apache starts when you boot up the hardware. If that's the case you need to write an init script in /etc/init.d or /etc/rc.d/init.d (depending on your system). This page gives a decent primer on this:
http://www.philchen.com/2007/06/04/quick-and-dirty-how-to-write-and-init-script
I have a rails app with the whenever gem installed to setup cron jobs which invoke various rake tasks. For reasons unbeknownst to me, each rake task gets invoked twice at precisely the same time. So my db backup task backs up the db twice at 4:00am.
Inspecting crontab reveals correct syntax for all of the cron jobs, so I don't think this is an issue with the whenever gem not correctly configuring the cron jobs. Also confusing is that in both staging and production environments and can invoke tasks on the command line and they only run once.
Any thoughts on what would cause this? I'm at a complete loss troubleshooting wise.
The number of cron jobs that run depends on the number of application instances running in the server box. Are you have two instances of rails application running in the same server box?