Is there any way to find out whether particular rake task is currently running from Rails controller for example? I have an extensive rake task, which takes 5-6 hours to finish.
I need to see status of that rake task from frontend web interface, like:
Task "some operation" is running...
Also it would be nice to be able to hard stop / run that rake task from within frontend web interface.
If found Railscast devoted to it, but the method described there allows only to run rake task from controller, but not to stop/see its status.
If your job is taking 5-6 hours to complete, you must run it in background. To run long running jobs in background you can use one of these:
Resque
Sidekiq
And for tracking status of your jobs, you can use corresponding status tracking gems:
ResqueStatus
SidekiqStatus
Personally, I prefer sidekiq for its efficiency.
NOTE: All the above mentioned gems has good enough docs. Kindly refer to those.
Related
I have a rails module that processes some active record objects, only about 15-20 at a time, that I need to start off every two minutes.
I have tried to offload it to sidekiq (and sidekiq-cron), which works, but with the concurrency, created many race conditions and duplicate data.
I really just need a simple rake task cron for rails or maybe sinatra (as I would create a new sinatra app just to complete these tasks)
I either need to force sidekiq to process in a single thread or
have a "cron" job run a rake task or even the module directly
def self.process_events
events = StripeEvent.where(processed: false)
events = StripeServices.arrange_processing_order events
events.each do |event_obj|
StripeServices.new(event_obj).process_event_obj
end
end
thanks for any point in the right direction.
edited
sorry I wasnt very clear. pushing my module to sidekiq caused concurrency issues that I wasnt ready for (my bit of code is not threadsafe), and with the restrictions that Heroku places on "crons", whats the best way to run a rake task every 2 min?
If Sinatra can do it, I would prefer it, but I cant find the solution for that same problem.
It's not clear what are you asking. You already tried option 1, you can try option 2 (create the task and cron it, it's pritty easy) and you'll know better than anyone if it's better.
Anyway, I guess that both methods will have concurrency problems if one task takes more than 2 minutes.
You can add extra flags to prevent two task to process the same ServiceEvent (maybe add a boolean "processing" and set it to true when a task takes it).
Or maybe you can have a lock file to prevent a task to run if another one is already running (you create a file with a specific location and name when the task starts and delete it when it finishes processing, you can check if the file exists before starting a new task).
I have rails app deployed over multiple instances and had too many rake tasks to run over different instances so it is hard to manage which rake tasks is already run or which one remaining.
is there any way to manage it from db side, as schema_migrations table managed by migrations. if yes then, i want know how migrations exactly works?.
any suggestions?.
Correct way: use deploy automation. Capistrano is a good choice. Then you'll never need to worry about things like running rake task
I think the rake tasks should have no side effects if you execute it multiple times. If the task is implemented that way, then there's no need to worry about which has been done and which is not.
I think if you want to get a status tracking for the Rake Task, a simple way is to implemented a model to record the execution status of the rake task, and update the model each time rake task is done.
You can use resque-scheduler(https://github.com/resque/resque-scheduler) to manage and track your tasks .
You can use Progress Bar gem to monitor the progress of a particular rake task.
And according to the above suggestion, automated deployment through capistrano is a good option. You can manage the rake tasks running sequence in the cap script.
I am not sure if this question is of the correct format for SO.
I have a rails app with deployment on Heroku. In development I am using the "crono" gem to send a simple email out every week to remind users of various things. I can see how to use this in production with Heroku. There is almost nothing on this in a google search. Therefore my question is: "what is the best way to implement a simple weekly job in a rails app that is deployed on Heroku. Heroku has "scheduler" but this is a "best effort" add on that can claim to be reliable (according to their documentation)
Thanks
There's two ways to achieve what you want:
1. Use the Heroku scheduler (I would)
Honestly it's just so simple to set up, I would use this. It's also extremely cheap because you only pay for the dyno while the job is running. After it's run Heroku destroys the dyno (it's a one off dyno)
The best way to implement it is to have you background jobs callable by a rake task, and simply have the scheduler call that.
If you have a time interval Heroku doesn't support, simply handle that in your code. For example if you want to send e-mails once a week, have a timestamp to record when the last email was sent. Run the scheduler once a day and just check to see if it's ok to send another email, if not do nothing.
2. Use some kind of scheduler gem
There's a bunch of them out there. For example, rufus.
In this case, you'd write your rufus jobs. You would then need to have a worker dyno always running for rufus, this is much more expensive.
You'd tell the dyno to run rufus and keep running rufus by specifying the command rufus needs in your procfile. Something like:
scheduler: rake rufus:scheduler # Add rake task for rufus scheduler process
(credit for the above snippet How to avoid Rufus scheduler being called as many times as there are no.of dynos?)
I am looking for the best way to run a very long task in Heroku.
I use Ruby On Rails for my web application and I have a very long task that I want to run it every week on Sunday during the night. It takes around 15~20 minutes. I already have Rufus-Scheduler, but I am not sure it is the most effective solution.
I also find something about Backgrounding Tasks in Heroku with Delayed Job. But is it the best way to handle it ?
Thanks.
This is what I use for a job that I run every night: https://devcenter.heroku.com/articles/scheduler
It works really well if your job is configured as a rake task. The guide at the link shows you how to configure everything and even addresses long-running jobs.
Heroku does not recommend to run long-running jobs with Heroku Scheduler.
Heroku says,
Scheduled jobs are meant to execute short running tasks or enqueue longer running tasks into a background job queue. Anything that takes longer than a couple of minutes to complete should use a worker dyno to run.
So, in my opinion, the best approach would be to use Heroku Scheduler to run rake task for every kind of job (short or long) but if a task takes longer than a couple of minutes then I would simply create a Background Job within that rake task. That way the scheduler will never run longer than a couple of minutes.
We're hosting our Ruby on Rails application with the EngineYard App Cloud, which has worked really well for us. They provide a series of deploy call backs (before_restart, after_restart, etc.) which are analogous to Capistrano's callbacks.
We have a series of Rake tasks within our application which maintain various parts of the application. If we add a new business rule to the database, we might need to reload the users' associated business rules, etc.
These are tasks that there's no hard or fast schedule for, but also we don't want to run each and every task on every deploy, because they slow down the deploy process.
Are there any systems which would allow us to define a task to be run on the next deploy, sort of like migrations. The ideal system in my mind would work as follows:
We realize that on the next deploy, a task will need to be run
We schedule the task via the system
On the next deploy, the system see the list of post-deploy tasks -- it notices that the most recent one has not been run on the specific server yet (as in how migrations notate the database when they're run so that only the most recent, unrun migrations are triggered) -- the new task is triggered
Any recommendations on best practices for scheduling these post-deploy tasks and have them fire off unless they've already been run on the server?
Thanks!
Try the after_party ruby gem which is modelled on the basic operation of db:migrate but is for post deployment tasks. Post deployment (rake) tasks are created with a name like so
lib/tasks/deployment/20130130215258_task_name.rake
You can of course call any ruby code from within the rake task. The documentation says it supports sync and async tasks (async tasks are long running tasks that you can have going on in the background while your app is starting up)
I've not used it but am about to give it a shot as we have similar requirements as you've described.
Two approaches come to my mind
Quick/dirty solution...could you just use migrations to do this? Create a Rails migration that fires off the tasks when rake db:migrate is run
Take the same approach as migrations. Create a peer table to the schema_migrations table, and then in your before_symlink.rb (or whereever else) run the tasks that have not been executed yet, and then update the table?
You should give rails_tasker a shot. It provides a straight-forward way to automating your post-deploy tasks. Here is an article that describes how to use the gem.