I have a rake task for use with Heroku Scheduler. My issue is, when I run the rake task on Heroku, the process never fully completes, instead it exits early.
What I am trying to do is perform an action on every member of an array. This array has 4,500 items. The function is like this on my Action model:
def my_rake_task(some_array)
i = 0
while i < some_array.count do
puts "#{i} number in {some_array[i]}"
i += 1
end
end
and is set up in scheduler.rake like:
desc "Run my rake task"
task :my_rake_task => :environment do
Action.my_rake_task
end
When I run this locally, everything is fine, and I'll see all 4,500 items in the array properly output. When I push to Heroku and check it via Heroku run rake my_rake_task it will output anywhere from ~1,000 to ~1,600 of the lines, and then stop.
My logs look like:
2015-01-17T16:44:28.259815+00:00 heroku[api]: Starting process with command `bundle exec rake my_rake_task` by my_user_name
2015-01-17T16:44:31.729533+00:00 heroku[run.8495]: Awaiting client
2015-01-17T16:44:31.765515+00:00 heroku[run.8495]: Starting process with command `bundle exec rake my_rake_task`
2015-01-17T16:44:32.155205+00:00 heroku[run.8495]: State changed from starting to up
2015-01-17T16:45:31.914275+00:00 heroku[run.8495]: State changed from up to complete
2015-01-17T16:45:31.896364+00:00 heroku[run.8495]: Process exited with status 0
I'm wondering if this could be because I'm only using one dyno? Or if there's some other reason? Any insight is much appreciated.
Based off the limited information that your provided I have a few solutions for you that can ensure your tasks run as intended.
First when you run rake tasks on Heroku, especially if they are long running tasks, run them detached. This will ensure that the connection to your terminal does not interfere with the task. For instance if you close your terminal before the task is complete it will end it prematurely.
heroku run:detached rake my_rake_task
Secondly if you are checking your logs and only see ~1600 lines of output, ensure that you are requesting more of the logs:
heroku logs -n 5000
Third you should be using a delayed_job https://github.com/collectiveidea/delayed_job
that will put everything in a queue, run tasks on worker dynos, have error reporting, and full logs for each task/job. You can view a rails cast on delayed job here http://railscasts.com/episodes/171-delayed-job
Related
I want to run both the database and migration in foreman. However, I found that they usually run at the same time when I start foreman. Since at the time I run migration the database has not fully started yet, it causes the migration to fail.
Heroku using Procfile could facilitate a release phase. The phase would be run after all the commands are run. Can I do the same using foreman in my computer?
Heroku does not rely to Procfile to maintain the release process. The Heroku build stack does.
Since the foreman provides us the way to run multiple processes at the same time, not running processes in order, so your problem is not the responsibility of foreman
However, you have some other ways to do so.
Simple: since foreman can start your process with shell command, you can use basic shell command sleep (in seconds) for delaying your process
db_process: start_db_script.sh
migrarion_process: sleep 5; bundle exec rake db:migrate --trace
Full control: Instead of run default migration rake task, you can write another rake task which check the connection to database before execute the migration ( refer to this answer)
retried = 0
begin
# Establishes connection
ActiveRecord::Base.establish_connection
# Try to reconnect
# It will raise error if cannot reach your database
ActiveRecord::Base.connection.reconnect!
Rake::Task["db:migrate"].invoke if ActiveRecord::Base.connected?
rescue => e
retried += 1
if retried <= 5 # Retry only 5 times
sleep 1 # Wait 1 seconds before retry
retry
end
puts "#{e} Cannot connect to your database with 5 seconds"
end
Task:
Delete an item in Async manner [HomeWork]
I have already configured Active jobs with delayed_jobs in my rails application but I am still confused about performing Async task in rails project.
Let's take an example:
I have some item to delete from the database but I want to do it in Async manner. I also read about perform_later or perform_now method in delayed_job blogs. Here is my code which is working fine:
Controller class
def destroy
PostJob.perform_now(params[:id])
respond_to do |format|
format.xml { head :ok }
format.js { render 'posts.js.erb' }
end
end
Job class
class PostJob < ActiveJob::Base
queue_as :default
def perform(id)
#post = Post.find(id)
#post.destroy
end
end
According to official doc of delayed_jobs I can add handle_asynchronously in the end of method to run in async manner. How can I implement in this case?
My Question:
When I am looking at destroy method it is not deleting the element in Async way. However every steps written in destroy method is in Synchronous. Am I wrong?
If it's not then How can I implement destroy method to delete post in async manner?
Backgrounding task and cron job are same thing?
Edit -1
After giving suggestion by A Fader Darkly, I changed perform_now to perform_later which is working perfectly for Async process but it is not deleting the entry from table (code is fine because it works when i user perform_now).
Also when I am running job manually by following command, Everything works fine:
rake jobs:work
Is there any way to execute delay_job task as soon as the queue get some new data?
If you change your destroy method to call:
PostJob.perform_later(params[:id])
it should happen asynchronously. If not, you have some more set-up to do.
For your questions:
Yes you are right, but what you say is a tautology. Everything in that method is synchronous - the job queue isn't used because of the perform_now. Thus destroy isn't deleting in an async way.
See above.
Cron jobs work on the operating system level and are scheduled regularly for particular times. You could have a cron job working every minute, for example, or every day, or week (on a particular day at a particular time). They run from a schedule file called a crontab.
'Backgrounding' a task simply stops it from taking over the IO of your terminal session. So you can carry on using the terminal while the process runs in the background. Generally this is done on an ad-hoc basis, so you don't have to wait for a heavy operation to complete before going on to do different tasks.
EDIT
Based on edits to the question, it sounds like Delayed Job daemon needs to be started. From the instructions:
Note: For Rails 4 replace script/delayed_job with bin/delayed_job
When running a queue locally, omit the 'RAILS_ENV=production' part of commands.
Running Jobs
script/delayed_job can be used to manage a background process which will start working off jobs.
To do so, add gem "daemons" to your Gemfile and make sure you've run rails generate delayed_job.
You can then do the following:
RAILS_ENV=production script/delayed_job start
RAILS_ENV=production script/delayed_job stop
Runs two workers in separate processes.
RAILS_ENV=production script/delayed_job -n 2 start
RAILS_ENV=production script/delayed_job stop
Set the --queue or --queues option to work from a particular queue.
RAILS_ENV=production script/delayed_job --queue=tracking start
RAILS_ENV=production script/delayed_job --queues=mailers,tasks start
Use the --pool option to specify a worker pool. You can use this option multiple times to start different numbers of workers for different queues.
The following command will start 1 worker for the tracking queue,
2 workers for the mailers and tasks queues, and 2 workers for any jobs:
RAILS_ENV=production script/delayed_job --pool=tracking --pool=mailers,tasks:2 --pool=*:2 start
Runs all available jobs and then exits
RAILS_ENV=production script/delayed_job start --exit-on-complete
or to run in the foreground
RAILS_ENV=production script/delayed_job run --exit-on-complete
For production purposes I need three processes running. This is my procfile and I use Foreman to start them:
web: bundle exec rails s Puma -p $PORT
queuing: bundle exec clockwork clock.rb
workers: bundle exec rake resque:workers
For deployment I'm using Mina. What's the appropriate way to start Foreman at the end of deploy task? Currently I'm starting like this:
desc "Deploys the current version to the server."
task :deploy => :environment do
deploy do
invoke :'git:clone'
invoke :'deploy:link_shared_paths'
invoke :'bundle:install'
invoke :'rails:db_migrate'
invoke :'rails:assets_precompile'
to :launch do
queue "touch #{deploy_to}/tmp/restart.txt"
queue "bundle exec foreman start"
end
end
end
... but I don't think that's the proper way since the "mina deploy" command never successfully exits and the local console just starts outputting whatever these processes are doing.
Question number two: How do I initialize logging for each of these three processes separately in separate files?
And how do I prevent killing all of these three processes when one of them crashes? How do I make the process restart when it crashes?
Thanks!
OK, so that's 3 questions.
1) I think you want to detach foreman process from the terminal. That way the deployment process will finish and foreman process will be running even after you have disconnected from the server. nohup is great for that, e.g. this will launch your app and pipe all logs to server.log file:
nohup foreman start > server.log 2>&1 &
2) AFAIK, foreman doesn't let you do that. You should probably use another process management service (e.g. systemd, upstart). Thankfully, foreman lets you easily export your config to different process management formats (http://ddollar.github.io/foreman/#EXPORTING).
3) Again, you probably want to separate your processes and manage them separately via upstart, systemd, etc.
I have a set of rake tasks that run on the production server, its detached from the main thread, and happens in the background
here is the code to execute it
def vehicle
#estate = Estate.find(#estate_id)
#date_string = #login_month.strftime("%m%Y")
system("rake udpms:process_only_vehicle[#{#date_string},#{#estate_id}] &")
redirect_to :controller => "reports/error_messages", :message => "Processing will happen in the background and reports will be refreshed after two minutes", :target => "_blank"
end
when this code is executed via the url route, it runs the rake task, i can see if i check the active processes on the production machine, but it ends abruptly after about 10 seconds.
ps axl | grep rake
this is the it shows
ruby /usr/local/rvm/gems/ruby-1.8.7-p352/bin/rake udpms:process_only_vehicle[082012,5]
if i execute the same same rake task in the app folder in the terminal it runs with out any errors. This runs without any issues on the dev machine. (OSX). Server is Mint. Rake version is the same on both. there is only one version of the gem.
since its the production server there are no logs (other than the produciton.log, and its no help). any help on how i go about debugging this issue will be much appreciated.
This is probably happening because your server software reaps requests that take longer than 10 seconds to respond. Despite the fact you're kicking off a rake task, it still has to wait for that system call to execute: if it takes awhile then the task will be terminated and the server worker returned to the worker pool.
In a more general sense, this is not the appropriate way to make a task happen in the background. You probably want to use Resque or Delayed Job, which enqueue tasks and run them in the background for you.
I'm using delayed_job to run jobs, with new jobs being added every minute by a cronjob.
Currently I have an issue where the rake jobs:work task, currently started with 'nohup rake jobs:work &' manually, is randomly exiting.
While God seems to be a solution to some people, the extra memory overhead is rather annoying and I'd prefer a simpler solution that can be restarted by the deployment script (Capistrano).
Is there some bash/Ruby magic to make this happen, or am I destined to run a monitoring service on my server with some horrid hacks to allow the unprivelaged account the site deploys to the ability to restart it?
For me the daemons gem was unreliable with delayed_job. Could be a poorly written script (was using the one on collectiveidea's delayed_job github page), and not daemons fault, I'm not really sure. But for whatever reason, it would restart inconsistently on deployments.
I read somewhere this was due to it not waiting for the process to actually exit, so the pid files would get overwritten or something. But I didn't really bother to investigate. I switched to the daemons-spawn gem using these instructions and it seems to be much more reliable now.
The delayed_job docs suggest that you use a monitoring service to manage the rake worker job(s). I use runit--works well.
(You can install it in the mode where it does not replace init.)
Added:
Re: restart by Capistrano: yes, runit enables that. Just do a
sudo sv kill delayed_job
in your Capistrano recipe to kill the delayed_job worker. Runit will then restart it with your newly deployed code base.
I have implemented small rake task that restarts the jobs task over and over again:
desc "Start a delayed_job worker in a endless loop to prevent exits."
task :jobs => :environment do
while true
begin
Delayed::Worker.new(:min_priority => ENV['MIN_PRIORITY'],
:max_priority => ENV['MAX_PRIORITY'],
:quiet => false).start
rescue Exception => e
puts "Exception occured (#{e})"
end
puts "Task jobs:work exited, clearing queue and restarting"
sleep 1
Delayed::Job.delete_all
end
end
Apparently it did not work. So I ended with this simple solution:
for (( ;; )); do rake jobs:work --trace; done
get rid of delayed job and use either whenever or resque