I have been operating under 0 workers, but need to increase the speed at which I'm processing background tasks. I am using Sidekiq for all of my background workers.
When I increase the worker dyno count to 1, I keep getting this error in my heroku logs:
dont know how to build task 'jobs:work'
From researching this, it seems like the issue is that heroku worker dynos are reliant on delayed_job and I am not using delayed_job anywhere.
If I install delayed_job, what will I have to change to get sidekiq to work? Or do I even need delayed_job?
Update your projects Procfile to specify sidekiq for the worker:
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec sidekiq
Then redeploy your application.
I think Heroku defaults to trying to run delayed_job if you don't specify the worker in your Procfile.
Related
I have a rails app using sidekiq, sidekiq-status, and sidekiq batching:
gem "sidekiq"
gem "sidekiq-status"
# freemium version vs sidekiq pro https://github.com/breamware/sidekiq-batch
gem "sidekiq-batch"
# slim & sinatra for sidekiq monitoring
gem "sinatra", "2.0.0.beta2", require: nil
Locally when I run sidekiq and redis, the jobs process.
When deployed to heroku, the jobs are queued but they do not process.
I am using the Rediscloud and have the env variable REDIS_PROVIDER set to REDISCLOUD_URL and both REDISCLOUD_URL and REDIS_URL set to the generated url from the Rediscloud addon.
Procfile:
high_worker: DB_POOL=$SIDEKIQ_DB_POOL bundle exec sidekiq -c $SIDEKIQ_CONCURRENCY -t 8 -q high
default_worker: DB_POOL=$SIDEKIQ_DB_POOL bundle exec sidekiq -c $SIDEKIQ_CONCURRENCY -t 8 -q default
low_worker: DB_POOL=$SIDEKIQ_DB_POOL bundle exec sidekiq -c $SIDEKIQ_CONCURRENCY -t 8 -q low
When I startup a queue manually on heroku, ie heroku run DB_POOL=10 bundle exec sidekiq -c 10 -t 8 -q low -a {my_app_name}, the queue processes those jobs.
What am I missing in my configuration?
Your Procfile defines process types, but it doesn't make them run.
Each process type can run on zero or more dynos. To change the number of dynos for each process type you can use the heroku ps:scale command. For example, to scale your low_worker process type to one dyno you can do
heroku ps:scale low_worker=1
Running dynos cost money and / or consume free dyno hours, so budget accordingly.
Alternatively, you can have your jobs run at scheduled times via the Heroku Scheduler. This is less appropriate for tasks that should run continuously, but it's an option.
I'm confused how heroku and sidekiq work. My Procfile looks like:
web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -e $RAILS_ENV
Now inside my rails I run my sidekiq jobs in my code like:
SomeWorker.perform_async(some.id)
Now will this automatically somehow make this process run in the worker dyno?
If yes, how does it just know to run this out of process?
It is confusing because when I am in my main git folder I can run heroku commands and I know this are for my web dyno, but how do I then see the logs for my worker dyno or will these be in my same dyno logs?
When you setup your Procfile, you're telling Heroku to setup 2 types of dynos: web and worker. It's likely that these are using the same Rails app code but are starting up with different commands (bundle exec puma vs. bundle exec sidekiq). You then can scale however many VMs (dynos) that you need for each type of process.
The glue that holds the two together is Redis. When you run SomeWorker.perform_async(some.id) from your web process, you're adding a record to Redis describing a job to run. Your worker process watches Redis for new records and then processes them.
The Heroku logs show logs from all running dynos. So you should see logs from both your web and worker processes mixed in together.
I was able to scale my Heroku workers by using heroku ps:scale worker=3 from CLI.
Do I also need to edit my Procfile like worker: env TERM_CHILD=1 QUEUE='*' COUNT='3' bundle exec rake resque:workers or is this redundant?
The command heroku ps:scale worker=3 will spin up 3 dynos each running the worker defined in your Procfile.
Your definition of worker is worker: env TERM_CHILD=1 QUEUE='*' COUNT='3' bundle exec rake resque:workers. This command will create 3 worker threads within a dyno. These worker threads will share the resources of the dyno (memory, cpu).
So if you make both changes then you'll end up with 3 dynos each with 3 worker threads - for a total of 9 workers threads.
Hope that clarifies things. I don't know the needs of your application I'll let you be the judge of that but if your jobs aren't that intensive then you may be able to get by with 1 dyno that has 3 worker threads. You can save some money this way. Goodluck
Whenever I run heroku run bundle exec sidekiq, I see all my background jobs being done, however, I want them to be able to go without me needing to be there. When I exit out of that terminal tab, sidekiq stops working. How would I mitigate that?
Also, I've read something about procfiles and increasing workers. I don't know what procfiles are and I don't know how to increase workers either.
Basically, I'm a newbie trying to get sidekiq set up to run on Heroku for my Rails app. I want it to be running at all times.
Create a file named ./Procfile with the following in it:
web: bundle exec rails server -p $PORT
worker: bundle exec sidekiq
sidekiq on Heroku
more on Procfiles
foreman gem
It's my first attempt to get Redis working on Heroku.
I've added one worker dyno (just today, so didn't pay yet), added RedisToGo Nano add-on, tested background jobs on my local machine, and pushed the app to heroku.
heroku ps
gives
=== web: `bundle exec rails server -p $PORT`
web.1: up 2013/03/03 18:26:09 (~ 37m ago)
=== worker: `bundle exec rake jobs:work`
worker.1: crashed 2013/03/03 19:02:15 (~ 1m ago)
Sidekiq Web Interface says that one job is enqueued, but zero processed or failed.
I'm guessing it's because my worker dyno is crashed.
Are there any noob mistakes that I don't know about?
(e.g. I need to run some command to start listening to background jobs etc)
heroku logs --tail doesn't show any errors, so I don't understand why my worker dyno chashes.
I did some research and fixed it like this:
Under app's root directory I created a file called "Procfile" with this content:
web: bundle exec rails server -p $PORT
worker: bundle exec sidekiq -c 5 -v
Got this idea from here.
After that it worked ok.
Also make sure you setup REDIS_PROVIDER:
heroku config:set REDIS_PROVIDER=REDISTOGO_URL
Sidekiq's GitHub page also has instruction. Click here