I usually just run "heroku run rake jobs:work" from the command line. The great thing about this apporach is that I get intimidate feedback on whether a job failed or not and what jobs are currently processing.
However, now I need to run "heroku ps:scale worker=1"
Is there a way to see what the worker is processing just like with the rake task via the command line?
You can use the heroku command like to access the log of the worker.
heroku logs -t --ps worker will show you what is currently being executed on your worker.
I also recommend to use gem workless which scales your worker up only when needed. This can save you a lot of money
You didn't say specifically what queue you are running, but it sounds like delayed_job. If so, you can install the delayed_job_web gem. Basic setup is as simple as adding it you your Gemfile and adding a route for it. Then you can browser to your site /delayed_job and manage the jobs.
Related
When I start sidekiq in my development environment (Rails 3.2), I use the following command:
bundle exec sidekiq
When I do this, sidekiq will execute all jobs that have been queued up when it was not running. e.g. If I have created a bunch of new user accounts during testing, it will try and send welcome emails to all of the fake accounts (my emails are sent from a sidekiq job).
Is there a way to start sidekiq and tell it to delete all pending jobs? That way I can turn it back on without worrying that it will try and run a bunch of jobs that don't need to run (since this is my dev environment).
I have looked in documentation, but can't find an answer, hopefully it's something simple I overlooked...
redis-cli flushall && bundle exec sidekiq
I found a solution: Using the sidekiq monitoring UI that comes with sidekiq (https://github.com/mperham/sidekiq/wiki/Monitoring), I'm able to view all queues (even when sidekiq is not running). Deleting the queue will remove all of the jobs in it, which solves the problem.
I recently set up the Scheduler add on and set up my rake task, 'rake cron_jobs:my_task'.
When I test it with
'heroku run rake cron_jobs:my_task', it works fine.
The scheduler also claims it ran when it was supposed to, and is scheduled to run again, but there's no logging associated with the process the way https://devcenter.heroku.com/articles/scheduler#inspecting-output says there should be.
'heroku ps' shows no scheduled dynos, 'heroku logs --ps scheduler.1' has no output.
What am I missing?
Actually I was trying to solve this myself, and did not find the answer anywhere, so here it is if someone else is struggling with this:
heroku logs --tail --ps scheduler
--tail is important to keep streaming the logs.
My best guess: the heroku ps and heroku logs commands only give you status logs for currently running processes/dynos.
So after the scheduled rake task is done, you can't reach the logs through the command line.
You can access the history of your logs by using one of the logging addons. Most of them offer a free tier too.
They all are based on the log drains which you also could use directly, if you want to build it yourself.
Here is what I do for that:
Simply in your tasks itself include put statements to know when the job started running and when it is finished as well.
Also, you can include puts statement in the executed job as well.
I'm using paper trial add-on which is a very powerful logging tool that you can search and find any particular log at a specific time. Also, you can add an alert when your schedule job started to run.
I had a similar problem (using the newer Heroku PGBackups Service) and found an unexpected explanation for it.
The rake task rake pgbackups-archive was not run by Heroku Scheduler, but it worked when I ran it manually from the command line.
In my case, I noticed that my issues were caused by the different time zone used by the Heroku interface (which seems not to be CET). So my rake task which should have run at a specific time daily effectively never ran, as I changed the specific time throughout the day for testing and I always missed the specified time in the Heroku timezone.
You can try running the task every ten minutes and see if it works.
It's nice to see Rails 4.2 come with Active Job as a common interface for background jobs. But I can't find how to start a worker in the document. It seems that the document is still immature (e.g. the right version of Sneakers is only referred to in Rails' Gemfile), so I'm not sure if the "running workers" part is not in Active Job or just not mentioned in docs.
So with Active Job, do I still need to manually start the job watcher threads like sidekiq or in my case, rake sneakers:run? If so, where should I put these commands to let rails server run these parallel tasks automatically in a develop environment?
ActiveJob is just a common interface. You still need the backend gem, and you still need to launch it separately from your server (it is a separated process, which is the objective).
Sample using resque:
In the Gemfile:
gem 'resque'
In the terminal, launching a worker:
bin/resque work
The case is similary when using sidekick, delayed job or something else.
If you want to launch the server & worker in a single command, you can create a short bash script for it, but I would advise not doing so: having two separated console helps to watch what is happening on each side (web app & worker).
A better solution would be to use the foreman gem to manage starting & stopping your process.
You can create a simple Procfile with the processes to start:
web: bundle exec rails s
job: bundle exec resque work
And then just start both using foreman:
foreman start
By default, foreman will interleave the logs of the process in the console, but this can be configured.
You still have to run the job thread watcher.
I want to run a rake script on my application on Heroku that will take several hours. If I start it from my console on my laptop, and shut down my laptop the script will stop (and I will have to shut down my laptop before the script is finished).
How do I do to start this rake script without it being "tied" to my laptop. I.e. so that it continues to run until it breaks or is finished?
heroku run rake update_feeds_complete --app myapp
is the script to run...
Giving me advice on what commands etc to Google would be helpful as well.
You can run commands with run:detached and close the terminal window and they will continue to run. So you simply need to run
heroku run:detached rake update_feeds_complete --app myapp
You might want to take a look at the scheduler, too. https://devcenter.heroku.com/articles/scheduler
Pretty handy for running rake tasks, even if one-offs.
In general, I use resque and redis for jobs I want to run on heroku. If it's a one-time job I may not, but if it's something I'm going to do regularly then having them installed is a great convenience.
redis installs through the 'Redis To Go' add-on with Heroku.
resque is a gem that's used for running background jobs of different kinds. There are lots of plugins for resque for sending email, for job scheduling, etc.
I'm not familiar with Heroku in particular, but in the past when i have run into this case I used remote execute via SSH:
ssh username#your.server -f 'your command here'
This will open an ssh session, execute your command and then close the connection. But your script will keep running remotely.
I finally got the DelayedJobs plugin working for Rails 2, and it does indeed work fine...as long as I run:
rake jobs:work
Just like the readme says, to be fair.
BUT, this doesn't fit my requirements...what kind of background task requires you to have a shell open, and a command running? That'd be like having to say script/server to run my rails app, and never getting that -d option so it'll keep running even after I close my shell.
Is there ANY way to keep the workers getting processed in the backgroun, or in daemon mode, or whatever?
I had a ray of hope when I saw the
You can also run by writing a simple
#script/job_runner#, and invoking it
externally:
Line in the readme...but...that just does the exact same thing the rake task does, you just call it a different way.
What I want:
I want to start my rails app, then start whatever will process the workers, and have BOTH of them run invisibly in the background, without the need for me to babysit it and keep the shell that started it running.
(My server is something I SSH into, so I don't want to have that shell that SSHed into it running 24/7 (especially since I like to turn off my local computer now and again)).
Is there any way to acomplish this?
You can make any *nix command run on the background by appending an & to its end:
rake jobs:work &
Just make sure you exit the shell (or use the disown command) to detach the process from your login session... Otherwise, if your session disconnects, the processes you own will be killed with it.
Perhaps Beanstalkd and Stalker?
Beanstalk is a fast and easy way to queue background tasks. Stalker provides a nice wrapper interface for creating these jobs.
See the railscast on it for more information
Edit:
You could also run that rake task as a cronjob which would mean the server would run it periodically without you needing to be logged in
Use the collectiveidea fork of delayed_job... It's more actively developed and has support for running the jobs in a daemon without any extra messing about.
My capistrano script calls
RAILS_ENV=production script/delayed_job start