I need some Rails application server for queuing some processes and run them in multiple threads simultaneously. I find Sidekiq too expensive for my purposes and I'm looking for an alternative. Can I run multiple threads with Resque? Should I use sucker_punch or Shoryuken? Thanks in advance!
-- UPDATED --
See answer in comments below.
Related
Here are some questions I have on ActiveJobs:
Say I've queued up n number of jobs on a job queue on sidekiq via ActiveJobs. On my EC2, I've set puma to have 4 workers, with 5 threads each. Does this mean up to 20 concurrent jobs will run at the same time? Will each thread pick up a queued job when it's idle and just process it? I tried this setting but it seems like it is still processing it in serial - 1 job at a time. Is there more settings I would need to do?
Regarding concurrency - how would I be able to setup even more EC2 instances just to tackle the job queue itself?
Regarding the queues itself - is there a way for us to manage / look at the queue from within Rails? Or should I rely on sidekiq's web interface to look at the queue?
Sidekiq has good Wiki. As for your questions:
Sidekiq(and other Background Job implementations) works as
where producer is your Rails app(s), Queue - Redis and consumer - Sidekiq worker(s). All three entities are completely independent applications, which may run on different servers. So, neither Puma nor Rails application can affect Sidekiq concurrency at all.
Sidekiq concurrency description goes far beyond SO answer. You can google large posts by "scaling Sidekiq workers". In short: yes, you can run separate EC2 instance(s) and set up Redis and tune Sidekiq workers count, concurrency per worker, ruby runtime, queues concurrency and priority and so so on.
Edited: Sidekiq has per worker configruration (usually sidekiq.yml). But number of workers is managed by system tools like Debian's Upstart. Or you can buy Sidekiq Pro/Enterprise with many features (like sidekiqswarm).
From wiki: Sidekiq API
Is there anything in its architecture that makes it hard to do?
I want to run an existing rails+sidekiq application in a VM with very little memory, and loading the entire rails stack in two different process is using a lot of RAM.
Puma is built to spin up homogenous web worker threads, and divide incoming requests among them. If you wanted to modify it to spawn off separate Sidekiq threads, it should technically be possible with a crazy puma.rb file, but there's no precedent I can find for doing so (edit: Mike’s answer below points out that the sucker_punch gem can essentially do this, for the same purpose of memory efficiency). Practically-speaking, if your VM cannot support running two Rails processes at a time, it probably won't be able to handle the increased memory load as your application does the work of both Sidekiq and Puma… but that depends on your workload.
If this is just for development purposes, you might be able to accomplish what you're looking for by turning on Sidekiq's inline mode (normally meant just for testing):
require 'sidekiq/testing'
Sidekiq::Testing.inline!
This will cause all perform_async calls to actually execute inline, instead of going into Redis and being picked up by the Sidekiq process.
Nothing official.
This is what sucker_punch is designed for.
As far as I know Observer pattern in Ruby on Rails is not made to be asynchronous meaning that Observer's execution will block the action being processed.
I know about delayed_job gem and I really like it but sometimes it looks a bit too heavy for certain purposes.
What about launching a new thread in the Observer's callback?
I spent some time trying to find pros and cons of such approach and failed.
So the question is: are there any serious drawbacks of Observer's threading?
Have you heard about sidekiq? It's the new "hot" gem to do background processing (vs resque or delayedjob).
From the FAQ:
sidekiq uses redis for storage and processes messages in a multi-threaded process.
It's just as easy to set up as resque but more efficient in terms
raw processing speed. Your worker code does need to be thread-safe.
There's also a railscast about it here.
I would recommend using that compared to creating your own thread.
DelayedJob and Sidekiq both present good options, and Rails offers full ActiveJob support now, here are the official docs - https://guides.rubyonrails.org/v4.2/active_job_basics.html
DelayedJob I think has been around the longest, created by the crew that built Shopify.com. It creates a table in your rails app and queues and works off of that. For me it provides the simplest option, as it carries no other dependencies other than your rails app.
Sidekiq offers a great alternative as well. It too is very simple and well-maintained, but instead of using your database, it uses a redis server to manage the jobs. That makes development a bit trickier, as you have to install redis and remember to start it when running your app. Not hard, just a bit extra stuff.
Here's a quick guide comparing the two - DelayedJob vs. Sidekiq, hope that helps.
I have 16 resque queues and when I try to see the memory allocaton for these queues it is showing like 4% of the memory for each fo these queues. But at that time all these queues are empty. SO, out of 100% of my memory nearly 64% is utilized by the environment load itself. Thats what I feel.
My doubt are
1. Will each of these resque queues loads the complete application into memory separately.
If Yes, can I make any change to the resque configuation in such a way that all resque queues use the same environment loaded in a single place in memory.
Thanks in advance
I think you are out of luck if you're using Resque. I believe this is why Sidekiq was developed as a nearly drop-in replacement for Resque. The author of Sidekiq wrote a blog post describing how he improved Resque's memory usage. Here's a little bit from the Sidekiq FAQs:
Why Sidekiq over Multi-threaded Resque?
Back at Carbon Five I worked on improving Resque to use threads
instead of forking. That project was the basis for Sidekiq. I would
suggest using Sidekiq over that fork of Resque for a few reasons:
MT Resque was a one-off for a Carbon Five client and is not supported.
There are a number of bugs that were not solved, e.g. the web UI's
display of worker threads, because they were not important to the
client.
Sidekiq was built from the ground up to use threads via
Celluloid.
Sidekiq has middleware, which lets you do cool things in
the job lifespan. Resque doesn't support middleware like this
natively.
In short, MT Resque: a quick hack to save one client a lot of money, Sidekiq: the well designed solution for the same problem.
I would like to write a Node.js UDP server on Heroku and plan to queue up the data to a Rails instance (dyno) for it to process? What are the pros and cons of using Delayed Job vs RabbitMQ? Thanks, Chirag
These are very hard to compare! RabbitMQ is a messaging system, while delayed_job is a database-backed task queue.
With RabbitMQ you can create a task queue, but that is just one of many use cases.
One could say that delayed_job is a very limited implementation of a task queue,, as the database is not suited for this kind of work.
(see e.g. http://www.rabbitmq.com/resources/RabbitMQ_Oxford_Geek_Night.pdf)
The database may work good enough for simple setups, but it is likely to eventually
fall apart.
If you want a task queue, I suggest you look for one that supports RabbitMQ.