Question:
Is there a way to see all of the running sidekiq instances for a single rails app? I want to know if there is another deployed instance with a sidekiq process picking up my jobs.
Backstory:
I'm in a situation where I need to know if another sidekiq process (on a new instance) has started picking up jobs so that I can enable some sort of pause mechanism to keep my current sidekiq process from doing anything
This has to do with blue-green deployment, and the problem is that jobs are at times being processed by the old workers during deployment while both machines are active.
This is the piece I was missing, the 'remote control' feature of sidekiq.
https://github.com/mperham/sidekiq/wiki/API#processes
Related
We currently use delayed_job and rails to manage some long running jobs in our system. Some of these jobs take potentially hours to run, but we also like to deploy rather frequently, often many times a day. The problem with this setup is that we have to restart delayed_job during deployment to pick up code changes, so that any new jobs are processed with the latest code.
The solution we've arrived at is that for any job that needs to run for more than some small amount of time, we fork the delayed job so that it returns immediately, and the forked process handles the work. This way a deploy can restart all the delayed job processes, while the long-running 'job' keeps going until it's finished as an orphaned process.
We've looked at sidekiq, but it looks like we'd have the same issue there when trying to deploy new code.
Has anyone developed a solution they would recommend for dealing with long-running background processes that span multiple deployments?
I'm running a Rails app on Heroku, and I have defined a custom process type to perform some long-running jobs, really long-running, a job can easily take something about an hour or more. I know it's better to split it into some small chunks, but that's quite problematic for that task.
And the issue is that when I push a new version — Heroku restarts all the dynos (web, workers, long workers — everything). I wonder is it possible to restart only some process types, e.g. only the web dynos?
No, that isn't possible. The easiest and most scalable way around this would be to split your long-running jobs into smaller chunks.
That way, you would have a lot of very small jobs being processed very quickly. When your app is restarted, you would be able to restart your process, as it wouldn't stop a long-running job.
Alternatively, one-off dynos won't be restarted when your app is deployed.
Using the heroku api, you can programmatically boot one-off dynos. Using that, you could start a one-off dyno for each long-running job you need to process.
That job would be processed (for up to 24 hours, where it would be cycled), and you would be able to deploy your app without restarting it.
I have sucker_punch worker which is processing a csv file, I initially had a problem with the csv file disappearing when the dyno powered down, to fix that i'm gonna set up s3 for file storage.
But my current concern is whether a dyno powering down will stop my worker in it's tracks.
How can I prevent that?
Since sucker_punch uses a separate thread on the same dyno and does not use an external queue or persistence (the way delayed_job, sidekiq, and resque do) you will be subject to losing the job when your dyno gets rebooted or stopped and you'll have no way to restart the job. On Heroku, dynos are rebooted at least once a day. If you need persistence and the ability to retry a job in the event a dyno goes down, I'd say switch to one of the other job libraries:
https://github.com/collectiveidea/delayed_job
https://github.com/mperham/sidekiq
https://github.com/resque/resque
However, these require using a Heroku Addon. You can get a way with the free version but you will still have to pay for the extra worker process. Other than that you'd have to implement your own persistence and retrying by wrapping sucker_punch. Here's a discussion on adding those features to sucker_punch: https://github.com/brandonhilkert/sucker_punch/issues/21 They basically say to use Sidekiq instead.
I am currently using Resque for dealing with background jobs in my application. Now i have 5 different queues at present (it ll grow very fast). Each of them is doing works like updating Solr indexes, Real-time Notification ,scheduled newsletters, delayed emails & SMS. And currently am using Resque as a rails gem and running the Resque from rails environment.
Now i am planning to move Solr index update task and scheduled newsletters to different server since these two performing heavy operations. One approach is just copy the rails directory to the new server and run the Resque jobs from the rails environment. But i am not comfortable doing this.
Another one is creating a seperate rake app for resque tasks. But the problem is, both these tasks are heavily tied to rails models and rails templating. I am totally unsure how to proceed next.
Has anyone faced the similar problem, and how you architected the application?
We use rubber to provision our servers and deploy. It's a plugin for Capistrano that does role-based deployment. One of the roles is "resque_worker" and any machine with that role will start up resque-pool to start processing work.
But you can do this much more simply. Just deploy your application to two different machines. Resque is designed to allow workers on different machines. As long as your second machine can access your redis server, everything will work just fine.
Here's my simple ideal case scenario for when I'd like delayed job to run:
When the first application server (whether through mongrel or passenger) starts, it'll start my delayed job workers.
When the last running application server terminates, it'll kill all the delayed job workers.
The first part (starting) is doable, although I'm not sure what the "right" or "best" way to do it is. Just make a conditional (on process not already running) system call to delayed_job start?
The second part (terminating) -- well, I'm not sure if it is doable or not. Definitely have no idea how this effect could be accomplished.
Any thoughts or ideas?
Is there another way that you start/end delayed job workers that you think is best?
Side question:
The main questions above are for the production environment -- a more difficult case because there are multiple app servers running at the same time. Could the same thing be easily done in the development environment (where there's guaranteed to only be one application server, not a cluster of them) by forking a child process to run the delayed job workers that would always terminate when the parent terminates? How would I go about doing this?
You could definitely pull the terminating off with god.
Simply watch the app processes and god will fire a callback when they're all stopped.