We're hosting our Ruby on Rails application with the EngineYard App Cloud, which has worked really well for us. They provide a series of deploy call backs (before_restart, after_restart, etc.) which are analogous to Capistrano's callbacks.
We have a series of Rake tasks within our application which maintain various parts of the application. If we add a new business rule to the database, we might need to reload the users' associated business rules, etc.
These are tasks that there's no hard or fast schedule for, but also we don't want to run each and every task on every deploy, because they slow down the deploy process.
Are there any systems which would allow us to define a task to be run on the next deploy, sort of like migrations. The ideal system in my mind would work as follows:
We realize that on the next deploy, a task will need to be run
We schedule the task via the system
On the next deploy, the system see the list of post-deploy tasks -- it notices that the most recent one has not been run on the specific server yet (as in how migrations notate the database when they're run so that only the most recent, unrun migrations are triggered) -- the new task is triggered
Any recommendations on best practices for scheduling these post-deploy tasks and have them fire off unless they've already been run on the server?
Thanks!
Try the after_party ruby gem which is modelled on the basic operation of db:migrate but is for post deployment tasks. Post deployment (rake) tasks are created with a name like so
lib/tasks/deployment/20130130215258_task_name.rake
You can of course call any ruby code from within the rake task. The documentation says it supports sync and async tasks (async tasks are long running tasks that you can have going on in the background while your app is starting up)
I've not used it but am about to give it a shot as we have similar requirements as you've described.
Two approaches come to my mind
Quick/dirty solution...could you just use migrations to do this? Create a Rails migration that fires off the tasks when rake db:migrate is run
Take the same approach as migrations. Create a peer table to the schema_migrations table, and then in your before_symlink.rb (or whereever else) run the tasks that have not been executed yet, and then update the table?
You should give rails_tasker a shot. It provides a straight-forward way to automating your post-deploy tasks. Here is an article that describes how to use the gem.
Related
I have a rails module that processes some active record objects, only about 15-20 at a time, that I need to start off every two minutes.
I have tried to offload it to sidekiq (and sidekiq-cron), which works, but with the concurrency, created many race conditions and duplicate data.
I really just need a simple rake task cron for rails or maybe sinatra (as I would create a new sinatra app just to complete these tasks)
I either need to force sidekiq to process in a single thread or
have a "cron" job run a rake task or even the module directly
def self.process_events
events = StripeEvent.where(processed: false)
events = StripeServices.arrange_processing_order events
events.each do |event_obj|
StripeServices.new(event_obj).process_event_obj
end
end
thanks for any point in the right direction.
edited
sorry I wasnt very clear. pushing my module to sidekiq caused concurrency issues that I wasnt ready for (my bit of code is not threadsafe), and with the restrictions that Heroku places on "crons", whats the best way to run a rake task every 2 min?
If Sinatra can do it, I would prefer it, but I cant find the solution for that same problem.
It's not clear what are you asking. You already tried option 1, you can try option 2 (create the task and cron it, it's pritty easy) and you'll know better than anyone if it's better.
Anyway, I guess that both methods will have concurrency problems if one task takes more than 2 minutes.
You can add extra flags to prevent two task to process the same ServiceEvent (maybe add a boolean "processing" and set it to true when a task takes it).
Or maybe you can have a lock file to prevent a task to run if another one is already running (you create a file with a specific location and name when the task starts and delete it when it finishes processing, you can check if the file exists before starting a new task).
I have rails app deployed over multiple instances and had too many rake tasks to run over different instances so it is hard to manage which rake tasks is already run or which one remaining.
is there any way to manage it from db side, as schema_migrations table managed by migrations. if yes then, i want know how migrations exactly works?.
any suggestions?.
Correct way: use deploy automation. Capistrano is a good choice. Then you'll never need to worry about things like running rake task
I think the rake tasks should have no side effects if you execute it multiple times. If the task is implemented that way, then there's no need to worry about which has been done and which is not.
I think if you want to get a status tracking for the Rake Task, a simple way is to implemented a model to record the execution status of the rake task, and update the model each time rake task is done.
You can use resque-scheduler(https://github.com/resque/resque-scheduler) to manage and track your tasks .
You can use Progress Bar gem to monitor the progress of a particular rake task.
And according to the above suggestion, automated deployment through capistrano is a good option. You can manage the rake tasks running sequence in the cap script.
I am not sure if this question is of the correct format for SO.
I have a rails app with deployment on Heroku. In development I am using the "crono" gem to send a simple email out every week to remind users of various things. I can see how to use this in production with Heroku. There is almost nothing on this in a google search. Therefore my question is: "what is the best way to implement a simple weekly job in a rails app that is deployed on Heroku. Heroku has "scheduler" but this is a "best effort" add on that can claim to be reliable (according to their documentation)
Thanks
There's two ways to achieve what you want:
1. Use the Heroku scheduler (I would)
Honestly it's just so simple to set up, I would use this. It's also extremely cheap because you only pay for the dyno while the job is running. After it's run Heroku destroys the dyno (it's a one off dyno)
The best way to implement it is to have you background jobs callable by a rake task, and simply have the scheduler call that.
If you have a time interval Heroku doesn't support, simply handle that in your code. For example if you want to send e-mails once a week, have a timestamp to record when the last email was sent. Run the scheduler once a day and just check to see if it's ok to send another email, if not do nothing.
2. Use some kind of scheduler gem
There's a bunch of them out there. For example, rufus.
In this case, you'd write your rufus jobs. You would then need to have a worker dyno always running for rufus, this is much more expensive.
You'd tell the dyno to run rufus and keep running rufus by specifying the command rufus needs in your procfile. Something like:
scheduler: rake rufus:scheduler # Add rake task for rufus scheduler process
(credit for the above snippet How to avoid Rufus scheduler being called as many times as there are no.of dynos?)
I am trying to find out the best way to run scripts in the background. I have been looking around and found plenty of options, but many/most seem to have become inactive in the past few years. Let me describe my needs.
The rails app is basically a front-end to configure when and how these scripts will be run. The scripts run and generate reports and send email alerts. So the user must be able to configure the start times and how often these scripts will run dynamically. The scripts themselves should have access to the rails environment in order to save the resulting reports in the DB.
Just trying to figure out the best method from the myriad of options.
I think you're looking for a background job queuing system.
For that, you're either looking for resque or delayed_job. Both support scheduling tasks at some point in the future -- delayed_job does this natively, whereas resque has a plugin for it called resque_scheduler.
You would enqueue jobs in the background with parameters that you specify, and then at the time you selected they'll be executed. You can set jobs to recur indefinitely or a fixed number of times (at least with resque-scheduler, not sure about delayed_job).
delayed_job is easier to set up since it saves everything in the database. resque is more robust but requires you to have redis in your stack -- but if you do already it's pretty much the ideal solution for your problem.
I recently learned about Sidekiq, and I think it is really great.
There's also a RailsCast about it - Sidekiq.
Take a look at the gem whenever at https://github.com/javan/whenever.
It allows you to schedule tasks like cron jobs.
Works very well under linux, and the last commit was 14 days ago. A friend of mine used it in a project and was pretty satisfied with it.
edit: take a look at the gem delayed_job as well, it is good for executing long tasks in the background. Useful when creating a cron job only to start other tasks.
What is the preferred way to create a background task for a Rails application? I've heard of Starling/Workling and the good ol' script/runner, but I am curious which is becoming the defacto way to manage this need?
Thanks!
Clarification: I like the idea of Rake in Background, but the problem is, I need something that is running constantly or every 10 hours. I am not going to have the luxury of sitting on a web request, it will need to be started by the server asynchronous to the activities occurring on my site.
Ryan Bates created three great screencasts that might really help you:
Rake in Background
Starling and Workling
Custom Daemon
He talks about the various pros and cons for using each one. This should help you get started.
It depends on your needs.
Try out delayed_job, which was created by Tobi delayed_job (last updated 2011), a Shopify founder.
There are forks by DHH deleayed_job (last updated 2008), and collectiveidea delayed_job (last updated 20 days ago as of 6/28/2018).
I usually rely on cronjob scheduling as it gives the flexibility without having to write separate code to schedule it. Anything that can be executed from shell, can be scheduled! Be it any script (ruby / rake task / py / bash / any other you like), cronjob scheduling can be easily achieved.
If running on windows, one can use scheduled tasks
Hope this helps.
async_observer is the best. It doesn't do all kinds of dumb busy wait stuff or lose jobs on worker crashes like starling, no DB polling, etc... and it integrates into rails remarkably well.
I push tons of jobs through it and it pretty much doesn't care.
Most of the plugins that have been mentioned will do the job, but if all you need is a Rake task run on a set schedule, then there's really no need to start throwing more architecture at it.
Just add a cron job which executes
"cd /path/to/rails/app; RAILS_ENV=production rake run:my:task"
Why reinvent the wheel, when Unix like operating systems have been running tasks on a schedule for decades?
I have used the daemons plugin in the past.
While I don't know if it is becoming a standard, I have had great success with BackgroundRB. I have several workers, some are long running tasks triggered by a user action while others are started on a schedule.
Have a look at Taskr. It's basically like cron, but with a RESTful web interface. You can use it to schedule tasks to periodically connect to your Rails app and trigger arbitrary code (via the Taskr4rails plugin). It's meant to fit nicely into a system built around RESTful services, plus it can notify you if a task returns an error, fails to run, etc.