I have these Resque workers defined in my Procfile
resque1: env QUEUE=mailer rake resque:work
resque2: env QUEUE=calculation rake resque:work
But seems non of them gets created, is it because the process type is wrong? If yes, how can I create two resque workers for my rails app?
[Problem Solved]
The reason been heroku only give one free dyno and the rails app already used one. Once I scale the web dyno to 0, and scale my resque dyno to 1 instance, the queued job gets executed immediately.
Thanks!
Related
My application is processing jobs with Resque.enqueue
I am starting the worker with rake resque:work QUEUE='converter'
To my understanding, in order to start 2 workers to process 2 requests for the same queue concurrently, I can start another worker with rake resque:work QUEUE='converter' in another terminal.
Is there a simple option to start 2 workers to work concurrently on the same queue without using the resque-pool gem, and without having to type in rake resque:work QUEUE='converter' twice?
It's advised within the same Resque codebase to only use COUNT=2 while on development environment. To manage more than one Resque worker, you'd need something like https://github.com/nevans/resque-pool
You can specify the worker count when running the command, like so:
$ COUNT=2 QUEUE='converter' rake resque:workers
How many workers are initialized by default in Resque queues when the following command is executed?
QUEUE=(QUEUE NAME) rake environment resque:work
rake resque:work starts exactly one worker. If you want to start multiple workers (for example four workers) you need to run:
COUNT=4 rake resque:workers
When you start resque:workers without a COUNT or with a COUNT of 0 then no workers will be started.
You might want to have a look at the implementation of the resque's rake tasks.
I was able to scale my Heroku workers by using heroku ps:scale worker=3 from CLI.
Do I also need to edit my Procfile like worker: env TERM_CHILD=1 QUEUE='*' COUNT='3' bundle exec rake resque:workers or is this redundant?
The command heroku ps:scale worker=3 will spin up 3 dynos each running the worker defined in your Procfile.
Your definition of worker is worker: env TERM_CHILD=1 QUEUE='*' COUNT='3' bundle exec rake resque:workers. This command will create 3 worker threads within a dyno. These worker threads will share the resources of the dyno (memory, cpu).
So if you make both changes then you'll end up with 3 dynos each with 3 worker threads - for a total of 9 workers threads.
Hope that clarifies things. I don't know the needs of your application I'll let you be the judge of that but if your jobs aren't that intensive then you may be able to get by with 1 dyno that has 3 worker threads. You can save some money this way. Goodluck
New to Rails and very new to Delayed Jobs.
Got one that's supposed to be triggered after 5 minutes. I finally got it to work so that if I run
rake jobs:work
in my terminal, the job starts up and works correctly. If I CTRL-C and exit that action in my terminal, the delayed job stops working correctly. This is one thing on my local server and another on Heroku, where I have to start up the delayed job using
heroku run rake jobs:work
I looked into the new Heroku toolbelt and downloaded the gem they suggest for worker maintenance, foreman, but when I run "foreman start", I get this error
ERROR: procfile does not exist
I don't know what a procfile is, I'm afraid of breaking things after spending pretty much a day debugging my delayed_jobs actions, and I want to do this right to make sure it works instead of figuring out some hacky fix that breaks down later -- so I figured I should ask this question, however obnoxiously vague it may be.
Should I be using foreman for this? Or workless? (Saw that in another SO question). Where's my procfile? Should I do anything with it?
Thanks,
Sasha
You should be using a procfile to set up your Heroku processes, this is the standard method that Heroku uses to define and control the processes.
If you haven't utilised a procfile to this point everything will probably still work as Heroku adds some default processes when you push a Rails app, including both the web and worker processes. The default worker process is set to delayed job.
Foreman was developed in order to set up your local machine to use the same approach but, unlike the Heroku service, Foreman actually requires a procfile to be present to control the services that are started when Foreman is started as it doesn't know how to setup defaults.
I would suggest creating a procfile, placed in the root directory of your project, to ensure that your processes are set up and operating in the same manner on your local machine as on Heroku. If you want to mimic what Heroku sets up automatically you add the following to the procfile depending on whether you are using the Thin web server (which Heroku recommends) or not.
With Thin in your gemfile:
web: bundle exec thin start -R config.ru -e $RACK_ENV -p $PORT
worker: bundle exec rake jobs:work
Without a special web server (eg you are using webrick, the rails default):
web: bundle exec rails server -p $PORT
worker: bundle exec rake jobs:work
Once this file is in place you can run foreman on your local machine and it will start your web server and delayed_job workers automatically.
Running through this process will only impact starting delayed_job on the local machine. As you are running the exact same command bundle exec rake jobs:work as you are currently using there should be no impact on your dj actions in either locally or on Heroku. Obviously some testing is required to make suer this is actually the case.
Workless is designed to scale workers on Heroku so that you don't have to pay for them when there is no work available. It has no bearing on the procfile or defining how to actually start a dj worker process.
as far as I know, there are 2 version of delayed_job:
original(tobi's) https://github.com/tobi/delayed_job
collectiveidea's fork: https://github.com/collectiveidea/delayed_job
when using the collectiveidea version, you should start it as below:
# Runs two workers in separate processes.
$ RAILS_ENV=production script/delayed_job -n 2 start
I am not familiar with delayed_job on Heroku, please follow its instructions.
I have an app on Heroku running delayed jobs. However at the moment I have to start the job queue running with the terminal command:
heroku rake jobs:work
...but this means when I shut down my terminal the app's delayed job queue shuts down too.
Is there a way I can get Heroku to just always start and run delayed job in the background when the app starts up? Without having to run the command each time and without having it directly linked to my terminal shell?
Thanks very much.
Edit:
It's on the bamboo stack. Upping workers or running rake jobs:work , the delayed jobs runs for a while, but then the queue seems to just stop getting processed. There are no errors in the delayed jobs queue, the workers just stop processing the jobs. It has to explicitly restarted every 5 or 10 mins.
From the docs:
On Heroku's Aspen or Bamboo stack, use heroku workers 1
On the Cedar stack, you put this line in your Procfile:
worker: bundle exec rake jobs:work
And then do heroku scale worker=1.
we use the workless gem with our heroku stack. it starts worker when the delayed_job queue > 0 and quits the worker when delayed_job queue goes to 0.
It turns out that I was using the wrong rake gem.
The following was causing an issues with rails 3 on Heroku:
gem 'rake', '0.9.2'
Updating the gem fixed the issues, even though there were not errors in the log:
gem "rake", "0.8.7"