I would like to know how many threads Puma is using. Is there a way to poll for this? Is there a supported way to then write this info to a log? What other info can I monitor?
Related
I have an app that has some operations that timeout every once in a while (given our Puma configured timeout) but given then process just dies and a new one comes over I have no way of knowing why/where the process was hung.
Is there a way for me to print all threads before Puma kills my process?
I've tried using on_worker_shutdown but that doesn't seem to be called on a timeout kill. This is a Rails 4.2 app running on Ruby 2.2.7.
You can try add a middleware that implements a time-out lower than the Puma and on this one you dumpy whatever you need/want.
It is not answering your question about Puma but might be a workaround to solve the issue that you have now.
Here are some questions I have on ActiveJobs:
Say I've queued up n number of jobs on a job queue on sidekiq via ActiveJobs. On my EC2, I've set puma to have 4 workers, with 5 threads each. Does this mean up to 20 concurrent jobs will run at the same time? Will each thread pick up a queued job when it's idle and just process it? I tried this setting but it seems like it is still processing it in serial - 1 job at a time. Is there more settings I would need to do?
Regarding concurrency - how would I be able to setup even more EC2 instances just to tackle the job queue itself?
Regarding the queues itself - is there a way for us to manage / look at the queue from within Rails? Or should I rely on sidekiq's web interface to look at the queue?
Sidekiq has good Wiki. As for your questions:
Sidekiq(and other Background Job implementations) works as
where producer is your Rails app(s), Queue - Redis and consumer - Sidekiq worker(s). All three entities are completely independent applications, which may run on different servers. So, neither Puma nor Rails application can affect Sidekiq concurrency at all.
Sidekiq concurrency description goes far beyond SO answer. You can google large posts by "scaling Sidekiq workers". In short: yes, you can run separate EC2 instance(s) and set up Redis and tune Sidekiq workers count, concurrency per worker, ruby runtime, queues concurrency and priority and so so on.
Edited: Sidekiq has per worker configruration (usually sidekiq.yml). But number of workers is managed by system tools like Debian's Upstart. Or you can buy Sidekiq Pro/Enterprise with many features (like sidekiqswarm).
From wiki: Sidekiq API
I need some Rails application server for queuing some processes and run them in multiple threads simultaneously. I find Sidekiq too expensive for my purposes and I'm looking for an alternative. Can I run multiple threads with Resque? Should I use sucker_punch or Shoryuken? Thanks in advance!
-- UPDATED --
See answer in comments below.
As described here, I'm detecting that I've been forked by Phusion Passenger, and revive a background thread that is aggregating some data that will eventually get packaged and sent to a remote server after a set amount of time. But sometimes, before the thread wakes up from the sleep, the process disappears, and (according to my log messages, that report the PID when the thread wakes up), I never hear from it again. Any way to control or prevent this?
You shouldn't be creating threads within a Passenger hosted process. If Passenger doesn't think your process is busy servicing requests, it is free to shut it down without warning. Those background threads should be used only in the course of your request processing.
What you want is a background job processing facility like delayed_job to offload this.
I've read though the Backburner Tutorial but it kind of glosses over what workers are exactly, and it seems as though god is used for production environments, but all the other examples just show Backburner.work being used, which doesn't process asynchronously. So I can't figure how to do process jobs in my queue while I'm using WEBrick in development mode.
I will update the tutorial but the simplest way to process tasks is to just use the rake task:
rake backburner:work
This will process tasks as they come in right in your console. If you have dynamic queues or just want to process one queue you can specify the queues here:
QUEUES=newsletter-sender,push-message rake backburner:work
That should work in development.