Whenever I run heroku run bundle exec sidekiq, I see all my background jobs being done, however, I want them to be able to go without me needing to be there. When I exit out of that terminal tab, sidekiq stops working. How would I mitigate that?
Also, I've read something about procfiles and increasing workers. I don't know what procfiles are and I don't know how to increase workers either.
Basically, I'm a newbie trying to get sidekiq set up to run on Heroku for my Rails app. I want it to be running at all times.
Create a file named ./Procfile with the following in it:
web: bundle exec rails server -p $PORT
worker: bundle exec sidekiq
sidekiq on Heroku
more on Procfiles
foreman gem
Related
Suppose there's a task
rake startupscript
that should run whenever the app boots, how can we automate that on heroku?
I know there's a heroku scheduler but that will run the task every 10 minutes instead of just once at boot. I also know of the Procfile and believe this can be a solution, although I do not yet know how to implement (and probably more importantly, I don't want to risk breaking anything else that can be configured via a Procfile, e.g. webserver etc). A lot of the Procfile docs focus on using it to alter web servers rather than app level rake tasks.
How can a rake task be made to run at boot?
You can add something like this to Procfile before you start your application services
# Run pre-release-tasks here
release: bundle exec rails db:migrate
# Then run your application
web: bundle exec puma -t 5:5 -p ${PORT:-3000} -e ${RACK_ENV:-development}
Anything tagged as release will run before the startup script runs
https://devcenter.heroku.com/articles/release-phase
I'm confused how heroku and sidekiq work. My Procfile looks like:
web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -e $RAILS_ENV
Now inside my rails I run my sidekiq jobs in my code like:
SomeWorker.perform_async(some.id)
Now will this automatically somehow make this process run in the worker dyno?
If yes, how does it just know to run this out of process?
It is confusing because when I am in my main git folder I can run heroku commands and I know this are for my web dyno, but how do I then see the logs for my worker dyno or will these be in my same dyno logs?
When you setup your Procfile, you're telling Heroku to setup 2 types of dynos: web and worker. It's likely that these are using the same Rails app code but are starting up with different commands (bundle exec puma vs. bundle exec sidekiq). You then can scale however many VMs (dynos) that you need for each type of process.
The glue that holds the two together is Redis. When you run SomeWorker.perform_async(some.id) from your web process, you're adding a record to Redis describing a job to run. Your worker process watches Redis for new records and then processes them.
The Heroku logs show logs from all running dynos. So you should see logs from both your web and worker processes mixed in together.
I was watching the RailsCast on Sidekiq and had some questions:
1) Sidekiq handles tasks through threads instead of processes. What does this mean? Why does it save on memory?
2) Does the method inside the worker class need to have a "perform" method?
3) On the Sidekiq docs, it says:
Start sidekiq from the root of your Rails application so the jobs
will be processed:
bundle exec sidekiq
So if I'm running this on localhost, I can run bundle exec sidekiq. If I pushed up to Heroku, what do I do now? How do I run Sidekiq on Heroku?
4) I am not sure if my Sidekiq is working. I have this code:
def set_defaults
self.clicks = 0 if clicks.blank?
self.title = TitleWorker.perform_async(orig_url)
end
But TitleWorker.perform_asynch(orig_url) in testing just seems to return a string of numbers. What is going on? How do I fix this?
While I'm not super clear about the first question, I might be able to answer the other questions you have:
Yes. It needs a Perform method to start the worker process.
Heroku is just amazing. You don't have to do any special configuration on Heroku for running Sidekiq. All you have to do is, just make sure RedisToGo is installed, and add this to the Procfile:
worker: bundle exec rake jobs:start
sidekiq: bundle exec sidekiq -c 15 -v
I have been operating under 0 workers, but need to increase the speed at which I'm processing background tasks. I am using Sidekiq for all of my background workers.
When I increase the worker dyno count to 1, I keep getting this error in my heroku logs:
dont know how to build task 'jobs:work'
From researching this, it seems like the issue is that heroku worker dynos are reliant on delayed_job and I am not using delayed_job anywhere.
If I install delayed_job, what will I have to change to get sidekiq to work? Or do I even need delayed_job?
Update your projects Procfile to specify sidekiq for the worker:
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec sidekiq
Then redeploy your application.
I think Heroku defaults to trying to run delayed_job if you don't specify the worker in your Procfile.
New to Rails and very new to Delayed Jobs.
Got one that's supposed to be triggered after 5 minutes. I finally got it to work so that if I run
rake jobs:work
in my terminal, the job starts up and works correctly. If I CTRL-C and exit that action in my terminal, the delayed job stops working correctly. This is one thing on my local server and another on Heroku, where I have to start up the delayed job using
heroku run rake jobs:work
I looked into the new Heroku toolbelt and downloaded the gem they suggest for worker maintenance, foreman, but when I run "foreman start", I get this error
ERROR: procfile does not exist
I don't know what a procfile is, I'm afraid of breaking things after spending pretty much a day debugging my delayed_jobs actions, and I want to do this right to make sure it works instead of figuring out some hacky fix that breaks down later -- so I figured I should ask this question, however obnoxiously vague it may be.
Should I be using foreman for this? Or workless? (Saw that in another SO question). Where's my procfile? Should I do anything with it?
Thanks,
Sasha
You should be using a procfile to set up your Heroku processes, this is the standard method that Heroku uses to define and control the processes.
If you haven't utilised a procfile to this point everything will probably still work as Heroku adds some default processes when you push a Rails app, including both the web and worker processes. The default worker process is set to delayed job.
Foreman was developed in order to set up your local machine to use the same approach but, unlike the Heroku service, Foreman actually requires a procfile to be present to control the services that are started when Foreman is started as it doesn't know how to setup defaults.
I would suggest creating a procfile, placed in the root directory of your project, to ensure that your processes are set up and operating in the same manner on your local machine as on Heroku. If you want to mimic what Heroku sets up automatically you add the following to the procfile depending on whether you are using the Thin web server (which Heroku recommends) or not.
With Thin in your gemfile:
web: bundle exec thin start -R config.ru -e $RACK_ENV -p $PORT
worker: bundle exec rake jobs:work
Without a special web server (eg you are using webrick, the rails default):
web: bundle exec rails server -p $PORT
worker: bundle exec rake jobs:work
Once this file is in place you can run foreman on your local machine and it will start your web server and delayed_job workers automatically.
Running through this process will only impact starting delayed_job on the local machine. As you are running the exact same command bundle exec rake jobs:work as you are currently using there should be no impact on your dj actions in either locally or on Heroku. Obviously some testing is required to make suer this is actually the case.
Workless is designed to scale workers on Heroku so that you don't have to pay for them when there is no work available. It has no bearing on the procfile or defining how to actually start a dj worker process.
as far as I know, there are 2 version of delayed_job:
original(tobi's) https://github.com/tobi/delayed_job
collectiveidea's fork: https://github.com/collectiveidea/delayed_job
when using the collectiveidea version, you should start it as below:
# Runs two workers in separate processes.
$ RAILS_ENV=production script/delayed_job -n 2 start
I am not familiar with delayed_job on Heroku, please follow its instructions.