I have a table in my database with a list of emails to be sent, each at a specific time (precision down to the minute).
I'm on heroku, and I don't want to spend anything right now.. Is there a way to do this? The only way I thought was to create a deamon/cron somewhere else and make it call a private url every minute.. any other idea? Any way to have some background process or something that can handle this (on Heroku and without paying extra for addons..)?
thanks!
Heroku's free cron addon runs only once a day, so it is not suitable. Their paid cron addon runs only once an hour, so it is also not suitable. Running a daemon/cron elsewhere is a hack that will become problematic very quickly. It's fundamentally bad architecture.
Using delayed_job with a single Heroku Worker makes sense. Plus, delayed_job lets you specify exactly when each job should be run, down to a 5-second granularity. Yes, it is $36/mo to do this. But it frees you from doing things the wrong way. Plus, if you expect that you will not need the Worker most of the time, you can look into auto-scaling delayed_job on Heroku so the Worker is only turned on when you need it.
There is a whole bunch of free online services that would be more than happy to request your web page on a schedule that you set. You don't need to spend or code anything. Just Google :)
Related
I am developing a website using Ruby on Rails and I am doing a bit of rough planning. I can and have deployed rails websites before, just adding to the database and retrieving from database based on my use case, but this time around, its a bit different. I am adding to database but i will need the data to be processed on the server before the data is being sent back to the user or when he decides to retrieve it. What i do not get is how i am going to process the data on the server. I know this doesnt follow the normal pattern for asking questions, i would search for it with google except I dont know what I am looking for. A nudge in the right direction will do.
What I want to do exactly is have users register and click a button (request) which puts the users id in an array , what I need to do on the server is to randomly or not randomly connect two users based on some qualities, this program keeps running infinitely, such that the user can come back later to check if he has been connected with someone already.
This kind of logic typically belongs in the controller, or perhaps on the models. You should read the Rails docs, particularly on controllers: http://guides.rubyonrails.org/action_controller_overview.html
I think you may find a lot of benefit from running a background job for this that is constantly looking for matches. You could have an infinitely running Sidekiq process that is queued up with users. Then once one finishes, just fire it up again.
Or you could create a rake task that does a User.find_each and have it run again when the task finishes. But this would make things blocking if you end up having a lot of users. I'd recommend one job per user and just bloat the system with them. This way you can scale out both horizontally and vertically.
You'll want to learn about ActiveJob and Sidekiq to help achieve what you're looking for :). Sidekiq requires Redis which you'll also have to setup as well. I'd recommend the redis-rails gem to help with the integration.
To go off BenMorganIO's answer, I think this is a job for a background worker. This is a job that is processed in the background, so it doesn't slow up your app. A good example of this is firing off an email in the background.
There are primarily 3 gems I've seen for this:
delayed_job
Resque
Sidekiq (just celebrated 5 years!)
Those should point you in the right direction.
Is there a way to automate and schedule an API request with Rails? I'd like to make a request and save that information into my database, however I'd probably only need it to make a request every few hours for the most up to date info. It's a good amount of data, so I'd like to have it stored as opposed to making a request every time a user visits.
Is there some kind of rake task that I can set up to do this for me on a schedule (or an alternative for what I'm hoping to accomplish)?
Thank you!
The clockwork or whenever gems are made for running scheduled tasks.
If you're hosted on Heroku, you can also use the Heroku Scheduler add-on to execute tasks in your Rails app every 10 minutes, every hour or daily.
The Above solutions are good and may suit your needs. There are more sophisticated queuing and scheduling solutions based on Redis that support multiple retries, have monitoring interfaces etc.
I have had good experiences with Resque and https://github.com/resque/resque-scheduler
If you can be sure your code is threadsafe there is also Sidekiq https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs
I'd check out Sidekiq. It uses redis and therefore supports retries like errata mentioned.
https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs
I'm trying to determine the best way to go about doing something for a project I have where I rely on an external API/service which takes ~2.5-4 seconds for a reply.
Currently I'm using javascript to load the api/data after the DOM has loaded then jquery updates a partial on the page. Pretty as the loader I have is, it still locks up the server process, so I'd like to move it out into a Heroku worker using delayed_job or something else? And the info from the API is user specific and not something that could be in a cron job.
The data I'm pulling only needs to be updated every few hours and is recorded locally in the DB, so I'm guessing an all out web socket such as that provided by Pusherapp.com would be overkill?
I'm leaning towards polling using delayed_job and waiting for a status update to determine it's completeness. Has anyone done this with delayed_job? Hints or caveats?
Thanks
Yeah you can definitely do something like that with delayed_job... but ultimately it sounds like you need something cron-like for scheduling, right? Alternatively, can't you use cron on heroku to just run a rake task every couple of hours?
I have a ruby on rails app that uses Heroku. I have the need to run things like import/export tasks on our db that lock up the whole system since they are so heavy on the DB. Is there a way to tell the system to only run these tasks when the database is not being used at that second?
There is no built-in way to schedule a job like this. There are a few things you can do, though.
Schedule the jobs to run during the least busy hours of the day. That will depend on your business, customer base and so on, but hopefully there is a window that is more suitable than others.
You could write your batch job to run for a longer time, doing small units of work. Between each unit of work, sleep for a few seconds, or take a look at the current load average and decide what to do based on that. This should lower the impact of the batch jobs.
Have the website update a "lock" somewhere, either in the database or in a memcached or something. If your normal website usage updates the database, you could look at the existing updated_at. Then only do batch work when there hasn't been any activity for a while. This doesn't guarantee that a new user won't pop in at the same time your batch job runs, of course, but could be a way to find a window where the site is less used.
Have you looked into using Background Jobs / Workers on Heroku? It's also worth reading about Heroku's Delayed Job queuing system
I have an application that checks a database every minute for any emails that are supposed to be sent out at that time. I was thinking about making this a rake task that would be run by a cron job every minute. Would there be a better solution to this?
From what I have read, this isn't ideal because rake has to load the entire rails environment every minute and this becomes expensive.
Thoughts?
Thanks.
You can use backgroundrb. This, however, will eat up memory away from your main Rails app as it will spawn one Ruby instance exclusive to backgroundrb.
You can also define a SystemController (or equivalent) in your main application, with various actions corresponding to the various household tasks your application should perform. You can "prod" it from crontab using wget or curl, the advantage being that it shares resources with your main application. Depending on how paranoid or you are, or on how vulnerable to DOS (or other types of attacks) exposing such a controller to, possibly, the outside world, you may choose to block access to this controller's URL from addresses other than the loopback (ideally in your reverse proxy, alternatively from the controller itself.)
One really simple method would be to have a script that does..
while true do
check_and_send_messages()
sleep 60
end
..which means you are not constantly respawning the Rails environment.
Obviously it has various flaws, but also has some benefits (for example, with your 1-Rake-per-minute, it the Rake task takes more than one minute, Rake will be running multiple times at once)
Also, the Railscasts episodes Rake in Background, Starling and Workling, and Custom Daemon might give you some ideas (they are describing exactly this task)
It turns out there's actually something built just for this: ar_mailer. ar_mailer queues up the e-mails into the DB and then sends them out periodically using the ar_mailer command. You can call ar_mailer every minute.
The nice thing about ar_mailer is that it basically requires very little change in terms of how you already send e-mails. You just need to inherit from ar_mailer instead of ActiveMailer. Using this method, you won't have to worry about running rake tasks in the background, forking processes, or anything like that - and in effect you get a real mail server with queued messages that are deleted when the mail is actually sent. This feature is important if you have a system that sends out large numbers of e-mail enmass. I've used ar_mailer to build a social network - so I can attest to its robustness.
Here's a good article that talks about ar_mailer in depth. I would strongly advise against rolling your own solution here as Eric has built a time-tested solution to this very problem.
I do what Vlad suggested (#2), with only local requests honored, and I'm paranoid enough to also require a specific query string tacked on to the url.
I have several periodic actions set up this way.